# ScreenPointToRay offset
**URL:** https://forum.magicleap.cloud/t/screenpointtoray-offset/2281
**Category:** Unity Development
**Tags:** development
**Created:** [July 26, 2023, 3:03pm UTC](https://forum.magicleap.cloud/t/screenpointtoray-offset/2281 "2023-07-26T15:03:54Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@T\_ReKT](https://forum.magicleap.cloud/u/T_ReKT)
#### Post date: [July 26, 2023, 3:03pm UTC](https://forum.magicleap.cloud/t/screenpointtoray-offset/2281/1 "2023-07-26T15:03:54Z")
Give us as much detail as possible regarding the issue you're experiencing.
**Unity Editor version** : 2022.2.21f1
**ML2 OS version** : 1.2.0
**MLSDK version** : 1.2.0
**Host OS** : (Windows/MacOS) both
Hi,
A scene has been setup with WebRTC and Meshing. Webrtc is transmitting the video and am trying to place game objects remotely. We transmit the click location on the video from remote, and then use following code to place game object.
```auto
Ray newTouchRay = Camera.main.ScreenPointToRay(touchPosition);
if(Physics.Raycast(newTouchRay, out RaycastHit hitInfo, 100, LayerMask.GetMask("Plane")))
{
// Code to place game object
}
```
There is an offset in the placement. The markers are having more offset as the clicks go further away from the centre of the screen. The offset is also in the direction away from centre of the screen.
---
### Author:  [@etucker](https://forum.magicleap.cloud/u/etucker)
#### Post date: [July 26, 2023, 6:04pm UTC](https://forum.magicleap.cloud/t/screenpointtoray-offset/2281/2 "2023-07-26T18:04:48Z")
Hi @T_ReKT,
Welcome to the Magic Leap 2 Forums. We are so grateful to have you here engaging with us.
> [@T\_ReKT](#):
>
> place game objects remotely.
Would you mind sharing a video? This will help me gain a better understanding of expected behavior.
If not, would you mind sharing the reproduction steps necessary to create this issue.
Best,
El
---
### Author:  [@T\_ReKT](https://forum.magicleap.cloud/u/T_ReKT)
#### Post date: [July 27, 2023, 8:29am UTC](https://forum.magicleap.cloud/t/screenpointtoray-offset/2281/3 "2023-07-27T08:29:50Z")
Hi,
Sure here is a video. We transmit the mouse click locations from webrtc video stream on a browser and place markers on device using above described method.
What we are noticing is as we move more towards the corners of the screen, the offset becomes larger.
Placing at the exact centre works on point.
> **[Untitled video.mp4](https://drive.google.com/file/d/1COAZxoNdQvOf_f2chYMuswHfW9isBPiz/view?usp=sharing)**
>
> Google Drive file.
---
### Author:  [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)
#### Post date: [July 27, 2023, 1:40pm UTC](https://forum.magicleap.cloud/t/screenpointtoray-offset/2281/4 "2023-07-27T13:40:30Z")
To make the clicks align to the real world, you will need to use the Camera's intrinsic and extrinsic values to convert pixel to world positions.
Here is an example that can be used with the CV camera image example:
> [@Localizing detected faces using ML2 transformation matrix?](https://forum.magicleap.cloud/t/localizing-detected-faces-using-ml2-transformation-matrix/2253/3):
>
> Unfortunately, the Unity forum might be a better place for this type of question since it is not specific to Magic Leap and out of scope for this forum. You may want to consider using the CVCamear pose for the origin of the ray. You can obtain it using the MLCVCamera.GetFramePose function. For example private void OnCaptureRawVideoFrameAvailable(MLCamera.CameraOutput capturedFrame, MLCamera.ResultExtras resultExtras, MLCamera.Metadata metadataHandle) { if (MLCVCamera.Get…
---
### Author:  [@T\_ReKT](https://forum.magicleap.cloud/u/T_ReKT)
#### Post date: [July 28, 2023, 10:18am UTC](https://forum.magicleap.cloud/t/screenpointtoray-offset/2281/5 "2023-07-28T10:18:56Z")
Hi,
I tried this solution, it seems to work for the corners and the centre, but anywhere else and the markers are placed with an offset. This time the offset is towards the centre of the screen rather than away tho.
> **[new Untitled video.mp4](https://drive.google.com/file/d/1m8AUxMJvZEbfAAFt70b07If-a4CiNMgw/view?usp=sharing)**
>
> Google Drive file.
The cyan markers are placed exactly the way its in the script (4 in the corners and one in centre of screen). An extra cyan marker is placed at the touch point as well. The brown markers are placed by using Physics.Raycast using the "worldRay".
---
### Author:  [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)
#### Post date: [July 28, 2023, 4:09pm UTC](https://forum.magicleap.cloud/t/screenpointtoray-offset/2281/6 "2023-07-28T16:09:24Z")
I have not been able to reproduce that issue on my end. The points will appear to no be aligned properly if the depth value does not match the actual depth of the object you are recasting against. I would recommend changing the function and using a physics raycast that would cast on the mesh generated by Magic Leap.
If you want more precise alignment, I recommend expanding the function to take into account the distortion coefficient.
---
### Author:  [@T\_ReKT](https://forum.magicleap.cloud/u/T_ReKT)
#### Post date: [July 29, 2023, 6:42am UTC](https://forum.magicleap.cloud/t/screenpointtoray-offset/2281/7 "2023-07-29T06:42:05Z")
The brown arrows in video above are from physics raycast.
```auto
using UnityEngine;
using UnityEngine.XR.MagicLeap;
public class SimpleCameraRaycast : MonoBehaviour
{
public Vector3 CastRayFromPixelToWorldPoint(int width, int height, Vector2 pixelPosition, MLCameraBase.IntrinsicCalibrationParameters parameters, Matrix4x4 cameraTransformationMatrix, float depth)
{
// Step 1: Normalize the image coordinates
Vector2 normalizedImagePoint = new Vector2(
(pixelPosition.x - parameters.PrincipalPoint.x) / width,
(pixelPosition.y - parameters.PrincipalPoint.y) / height);
// Account for aspect ratio
float aspectRatio = width / (float)height;
// Account for FOV
float verticalFOVRad = parameters.FOV * Mathf.Deg2Rad;
float horizontalFOVRad = 2 * Mathf.Atan(Mathf.Tan(verticalFOVRad / 2) * aspectRatio);
normalizedImagePoint.x *= Mathf.Tan(horizontalFOVRad / 2);
normalizedImagePoint.y *= Mathf.Tan(verticalFOVRad / 2);
// Step 2: Convert normalized image coordinates to camera coordinates
Vector3 cameraPoint = new Vector3(normalizedImagePoint.x, normalizedImagePoint.y, 1);
// Step 3: Create a 3D ray from camera position in the direction of the camera coordinate
Ray cameraRay = new Ray(Vector3.zero, cameraPoint);
// Step 4: Convert the ray to world coordinates
Quaternion rotation = cameraTransformationMatrix.rotation;
Vector3 position = cameraTransformationMatrix.MultiplyPoint(cameraRay.origin);
Vector3 direction = rotation * cameraRay.direction;
Ray worldRay = new Ray(position, direction);
// Return the point in world space at the specified depth along the ray
return worldRay.GetPoint(depth);
}
}
```
In this function i return the world ray and perform the raycast on it.
One thing I didn't really understand is the "1" in `Vector3 cameraPoint = new Vector3(normalizedImagePoint.x, normalizedImagePoint.y, 1);`
It seems to also change the position of the markers placed.
I am performing raycast on the `worldRay`
---
### Author:  [@T\_ReKT](https://forum.magicleap.cloud/u/T_ReKT)
#### Post date: [July 29, 2023, 8:34am UTC](https://forum.magicleap.cloud/t/screenpointtoray-offset/2281/8 "2023-07-29T08:34:28Z")
Another thing is, that the stream is from the Main camera, and pose is from the CV camera. Will that be causing issues?
If so, don't think there is any way to capture mixed reality content in CV camera. Is it possible to get the transformation matrix of main camera? I could not find such api's in the docs.
---
### Author:  [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)
#### Post date: [August 14, 2023, 4:55pm UTC](https://forum.magicleap.cloud/t/screenpointtoray-offset/2281/9 "2023-08-14T16:55:39Z")
@T_ReKT Yes, you can get the camera information from both main and CV streams:
> **[Intrinsic/Extrinsic Parameters | MagicLeap Developer Documentation](https://developer-docs.magicleap.cloud/docs/guides/unity/camera/ml-camera-metadata)**
>
> This section includes details on reading the Intrinsic and Extrinsic parameters from the Magic Leap camera. These values can be queried using the MLCamera.ResultExtras value provided in the Camera Capture callbacks.
The "1" in the camera point vector is the direction of the ray with the Z axis being the forward direction. Then we transform the direction from local to world space relative to the CV camera pose.
---
### Author:  [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)
#### Post date: [August 21, 2023, 10:47pm UTC](https://forum.magicleap.cloud/t/screenpointtoray-offset/2281/10 "2023-08-21T22:47:27Z")
I have filed a bug ticket regarding this issue. I have been able to reproduce the issue you mentioned when using Mixed Reality Capture.
---
### Author:  [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)
#### Post date: [September 4, 2023, 3:53pm UTC](https://forum.magicleap.cloud/t/screenpointtoray-offset/2281/11 "2023-09-04T15:53:07Z")
There seems to be an issue with the original script. I have updated the script in the previous post with the correct function. See the Camera Utility script that has been added:
> [@Localizing detected faces using ML2 transformation matrix?](https://forum.magicleap.cloud/t/localizing-detected-faces-using-ml2-transformation-matrix/2253/3):
>
> Unfortunately, the Unity forum might be a better place for this type of question since it is not specific to Magic Leap and out of scope for this forum. You may want to consider using the CVCamear pose for the origin of the ray. You can obtain it using the MLCVCamera.GetFramePose function. For example private void OnCaptureRawVideoFrameAvailable(MLCamera.CameraOutput capturedFrame, MLCamera.ResultExtras resultExtras, MLCamera.Metadata metadataHandle) { if (MLCVCamera.Get…