# CV mapping with Depth Camera

**URL:** <https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/5198>\
**Category:** Unity Development\
**Tags:** depth-camera, sensors\
**Created:** [February 13, 2025, 2:41pm UTC](https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/5198 "2025-02-13T14:41:11Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![psyje6](https://avatars.discourse-cdn.com/v4/letter/p/ecd19e/32.png) [@psyje6](https://forum.magicleap.cloud/u/psyje6)\
**Post date:** [February 13, 2025, 2:41pm UTC](https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/5198/1 "2025-02-13T14:41:11Z")

</div>

**Unity Editor version** : 2022.3.42f1  
**ML2 OS version** : 1.8.0  
**Unity SDK version** : 2.5.0  
**Host OS** : MacOS

Hi, I am working on a project where I would like to use the 2D coordinates from my object detection model and convert them to world space using the depth camera. But I’m a bit confused on where to start, as the other posts here seem to be using a different API then is in the current documentation. Could you please point me in the right direction. Thank you!

Example:

> **[Depth Camera Example | MagicLeap Developer Documentation](https://developer-docs.magicleap.cloud/docs/guides/unity-openxr/pixel-sensor/depth-camera-example/)**
>
> This section includes examples on how to configure the Depth Camera Pixel Sensor and display it's data.

> [@CV mapping with Depth Camera](https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/3297/4):
>
> We do not have this type of API at this time, but I can share your feedback with our voice of customer team. One thing that you need to keep in mind when trying to cast a ray from the CV camera and get a hit from the depth image is that the CV camera and depth camera are not in the same place and do not have the same intrinsics or distortion coefficients. There are a couple of different ways one could approach this - for example, the azure kinect api has a set of transform function that perfor…

---

<div class="post-metadata">

**Author:** ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)\
**Post date:** [February 13, 2025, 4:45pm UTC](https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/5198/2 "2025-02-13T16:45:16Z")

</div>

I think the other link was in reference to how another device aligns the data from the two sensors, not that it is specific to the device. You are correct to use the Pixel Sensor API.

We do not provide SDK functionality to align or synchronize sensors, so this would need to be done using custom logic in your application. Note that the RGB camera and the Depth camera are not located at the same place on the Magic Leap Headset. So you will need to use the Intrinsic and Extrinsic data to align the RGB and Depth images.

---

<div class="post-metadata">

**Author:** ![psyje6](https://avatars.discourse-cdn.com/v4/letter/p/ecd19e/32.png) [@psyje6](https://forum.magicleap.cloud/u/psyje6)\
**Post date:** [February 17, 2025, 1:17pm UTC](https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/5198/3 "2025-02-17T13:17:23Z")

</div>

Thank you! That helped me make a good start. However I’m getting stuck when getting the pose of the frame. I’m doing it as soon as the camera callback is triggered as suggested in the other post. But I’m getting the same MLCVCameraGetFramePose error and I’m really confused as to why. Here is my function:

```auto
private void OnCaptureRawVideoFrameAvailable(MLCameraBase.CameraOutput cameraOutput,
            MLCameraBase.ResultExtras resultExtras,
            MLCameraBase.Metadata metadata)
        {

            MLResult result = MLCVCamera.GetFramePose(resultExtras.VCamTimestamp, out Matrix4x4 outMatrix);
            if (result.IsOk)
            {
                string cameraExtrinsics = "Camera Extrinsics";
                cameraExtrinsics += "Position " + outMatrix.GetPosition();
                cameraExtrinsics += "Rotation " + outMatrix.rotation;

                CameraExtrinsics.position = outMatrix.GetPosition();
                CameraExtrinsics.rotation = Matrix4x4.Rotate(outMatrix.rotation);
                CameraExtrinsics.timestamp = resultExtras.VCamTimestamp;

                Debug.Log(cameraExtrinsics);
            }

            CameraIntrinsics.fx = resultExtras.Intrinsics.Value.FocalLength.x;
            CameraIntrinsics.fy = resultExtras.Intrinsics.Value.FocalLength.y;

            CameraIntrinsics.cx = resultExtras.Intrinsics.Value.PrincipalPoint.x;
            CameraIntrinsics.cy = resultExtras.Intrinsics.Value.PrincipalPoint.y;

            UpdateRGBTexture(ref videoTexture, cameraOutput.Planes[0]);
}

```

---

<div class="post-metadata">

**Author:** ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)\
**Post date:** [February 17, 2025, 4:12pm UTC](https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/5198/4 "2025-02-17T16:12:43Z")

</div>

If you are using OpenXR , you will need to make sure the Perception Snapshots is enabled in the OpenXR Magic Leap Support Feature in your project settings

> **[MLCamera | MagicLeap Developer Documentation](https://developer-docs.magicleap.cloud/docs/guides/unity-openxr/ml-camera/#enabling-perception-snapshots)**
>
> The Magic Leap 2 MLCamera API allows developers to capture real and virtual content inside their applications. While the Magic Leap 2 has only one camera for capturing content, two separate streams can be access from the camera at the same time. This...

---

<div class="post-metadata">

**Author:** ![psyje6](https://avatars.discourse-cdn.com/v4/letter/p/ecd19e/32.png) [@psyje6](https://forum.magicleap.cloud/u/psyje6)\
**Post date:** [February 17, 2025, 8:34pm UTC](https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/5198/5 "2025-02-17T20:34:57Z")

</div>

I had enabled it before but am still getting the same error.

---

<div class="post-metadata">

**Author:** ![psyje6](https://avatars.discourse-cdn.com/v4/letter/p/ecd19e/32.png) [@psyje6](https://forum.magicleap.cloud/u/psyje6)\
**Post date:** [February 18, 2025, 11:20am UTC](https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/5198/6 "2025-02-18T11:20:29Z")

</div>

This is the specific error I am getting

```auto
leapcore/frameworks/perception/data_sources/include/pad/xpad_data_source.h(105) GetClosestTimestampedData():
ERR: Data Not Found for timestamp: 1022446282us, now time: 1045159819us
Error Unity Error: MLCVCameraGetFramePose in the Magic Leap API failed. Reason: MLResult_PoseNotFound 
Error Unity UnityEngine.XR.MagicLeap.MLResult:DidNativeCallSucceed(Code, String, Predicate`1, Boolean)
Error Unity UnityEngine.XR.MagicLeap.MLCVCamera:InternalGetFramePose(CameraID, MLTime, Matrix4x4&)
Error Unity YoloHolo.Services.MLImageAcquiringService:OnCaptureRawVideoFrameAvailable(CameraOutput, ResultExtras, Metadata)
Error Unity UnityEngine.XR.MagicLeap.Native.DispatchPayload3`3:Dispatch()
Error Unity UnityEngine.XR.MagicLeap.Native.MLThreadDispatch:DispatchAll()

```

---

<div class="post-metadata">

**Author:** ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)\
**Post date:** [February 18, 2025, 2:51pm UTC](https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/5198/7 "2025-02-18T14:51:01Z")

</div>

You will need to make sure to get the pose of the camera within 500ms of the callback. This means that you need to maintain a high frame rate, otherwise the timestamp might become invalid.

* * *

You can also use the OpenXR Pixel Sensor logic to obtain the camera pose. This way you don’t have to convert between OpenXR and MLSDK poses. Note, that if you are using the MLCamera , you can just use the following example without additional configuration. Just call GetSensorPose in the callback.

> **[API Overview | MagicLeap Developer Documentation](https://developer-docs.magicleap.cloud/docs/guides/unity-openxr/pixel-sensor/pixel-sensor-api-overview/#example-1)**
>
> This extension allows developers to interact with the Magic Leap 2's pixel sensors within their applications. It provides APIs for managing sensor data acquisition from various sensor types, each with unique capabilities and requirements. The...

Callback example

```csharp

    private MagicLeapPixelSensorFeature _pixelSensorFeature;
    private PixelSensorId _sensorType;
    private XROrigin _xrOrigin;
...

    void RawVideoFrameAvailable(MLCamera.CameraOutput output, MLCamera.ResultExtras extras, MLCameraBase.Metadata metadataHandle)
    {
        if (output.Format == MLCamera.OutputFormat.RGBA_8888)
        {
            //Flips the frame vertically so it does not appear upside down.
            MLCamera.FlipFrameVertically(ref output);
            UpdateRGBTexture(ref _videoTextureRgb, output.Planes[0], _screenRendererRGB);
        }

        if(_pixelSensorFeature == null)
        {
            _pixelSensorFeature = OpenXRSettings.Instance.GetFeature<MagicLeapPixelSensorFeature>();
            _sensorType = _pixelSensorFeature.GetSupportedSensors().Find(x=> x.SensorName == "Picture Center"); // Simplified for example

            bool wasCreated = _pixelSensorFeature.CreatePixelSensor(_sensorType);
        }

        Pose sensorPose = _pixelSensorFeature.GetSensorPose(_sensorType);

        // Updates the Sensor Pose to be relative to the XR Origin
        if (_xrOrigin = FindAnyObjectByType<XROrigin>())
        {
            Vector3 worldPosition = _xrOrigin.CameraFloorOffsetObject.transform.TransformPoint(sensorPose.position);
            Quaternion worldRotation = _xrOrigin.transform.rotation * sensorPose.rotation;
            // Update the existing pose
            sensorPose = new Pose(worldPosition, worldRotation);
        }

        Debug.Log("Sensor Position:" + sensorPose.position);
        Debug.Log("Sensor Rotation:" + sensorPose.rotation);

    }

```

---

<div class="post-metadata">

**Author:** ![psyje6](https://avatars.discourse-cdn.com/v4/letter/p/ecd19e/32.png) [@psyje6](https://forum.magicleap.cloud/u/psyje6)\
**Post date:** [February 19, 2025, 10:03pm UTC](https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/5198/8 "2025-02-19T22:03:13Z")

</div>

> [@kbabilinski](#):
>
> ```auto
> f (_xrOrigin = FindAnyObjectByType<XROrigin>())
> {
> Vector3 worldPosition = _xrOrigin.CameraFloorOffsetObject.transform.TransformPoint(sensorPose.position);
> Quaternion worldRotation = _xrOrigin.transform.rotation * sensorPose.rotation;
> // Update the existing pose
> sensorPose = new Pose(worldPosition, worldRotation);
> }
> 
> ```

Hi thank you for this. What does it mean that the sensor pose is relative to the XR origin?  
I am using the MLDepthCamera like this, I’m assuming there’s no difference in what the PixelSensor and the MLDepthCamera return?:

```auto
public override void Update()
        {
            //Debug.Log("Depth camera update");
            if (!permissionGranted || !MLDepthCamera.IsConnected) return;

            var result = MLDepthCamera.GetLatestDepthData(0, out MLDepthCamera.Data data);
            isFrameAvailable = result.IsOk;
            if (isFrameAvailable)
            {
                lastData = data;

                DepthCameraIntrinsics.fx = data.Intrinsics.FocalLength.x;
                DepthCameraIntrinsics.fy = data.Intrinsics.FocalLength.y;
                DepthCameraIntrinsics.cx = data.Intrinsics.PrincipalPoint.x;
                DepthCameraIntrinsics.cy = data.Intrinsics.PrincipalPoint.y;
                DepthCameraIntrinsics.width = data.Intrinsics.Width;
                DepthCameraIntrinsics.height = data.Intrinsics.Height;

                DepthCameraExtrinsics.position = data.Position;
                DepthCameraExtrinsics.rotation = Matrix4x4.Rotate(data.Rotation);
                DepthCameraExtrinsics.timestamp = data.FrameTimestamp;

            }
        }

```

Also do you have any advice on aligning the image from the CV Camera and depth camera as I am struggling to.

---

<div class="post-metadata">

**Author:** ![psyje6](https://avatars.discourse-cdn.com/v4/letter/p/ecd19e/32.png) [@psyje6](https://forum.magicleap.cloud/u/psyje6)\
**Post date:** [February 20, 2025, 10:29am UTC](https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/5198/9 "2025-02-20T10:29:28Z")

</div>

I was also wondering if you could explain how to access the depth information from the MLDepthCamera.FrameBuffer depthFrame. I’m currently converting it into an array to get the depth value at each pixel, but my depth values seem to be too high.

---

<div class="post-metadata">

**Author:** ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)\
**Post date:** [February 20, 2025, 4:36pm UTC](https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/5198/10 "2025-02-20T16:36:46Z")

</div>

Regarding the Pose:

When using the MLSDK APIs with OpenXR , you will need to set the App into Unbounded tracking space. As mentioned here : [MLCamera | MagicLeap Developer Documentation](https://developer-docs.magicleap.cloud/docs/guides/unity-openxr/ml-camera/#enabling-perception-snapshots)

This is because the MLSDK returns a Pose in a diferent space which is the equivalent of OpenXR’s unbounded.

* * *

Regarding the Depth Camera API:

Please note, the MLSDK API (MLDepthCamera) has been deprecated in favor of the OpenXR Magic Leap Pixel Sensor API. This means that the API will not receive updates and is no longer supported. That said, here is an old post regarding the ML Depth Camera:

> [@Processing the depth frames](https://forum.magicleap.cloud/t/processing-the-depth-frames/3284/13):
>
> Here is a script that sends an array of world points to a "point cloud renderer" . I have not included the point cloud renderer code but the script should be enough to demonstrate how to obtain the undistorted depth data. A few things to note. You may want to check if the depth point falls into the user's field of view and exclude any depth points that are not visible. This will ensure that the depth image aligns to the world more closely. Although, the point cloud is undistorted, some skewing …

---

<div class="post-metadata">

**Author:** ![psyje6](https://avatars.discourse-cdn.com/v4/letter/p/ecd19e/32.png) [@psyje6](https://forum.magicleap.cloud/u/psyje6)\
**Post date:** [March 25, 2025, 11:04am UTC](https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/5198/11 "2025-03-25T11:04:14Z")

</div>

Thank you so much for your help. I’ve been trying to get this working for the past month but am having issues aligning the two images. Would you be able to tell me if this approach is along the right tracks? It follows after the point cloud has been generated.

```auto
 // This method converts YOLO detection RGB coordinates to world coordinates based on depth camera's parameters.
    public static Vector3 GetWorldPointFromDepthImage(Vector2 yoloPixel,
                                               Vector2Int rgbImageSize,
                                               Vector2Int depthImageSize,
                                               Vector3[] depthImageWorldPoints,
                                               MLDepthCamera.Data depthCameraData,
                                               MLCameraBase.ResultExtras rgbCameraData,
                                               Matrix4x4 rgbCameraExtrinsics)
    {
        // Step 1: Map the YOLO pixel coordinates from the RGB image to camera space (RGB Camera).
        Vector3 rgbCameraCoords = ConvertPixelToCameraCoordinates(yoloPixel, rgbImageSize, rgbCameraData);

        // Step 2: Convert from RGB Camera space to Depth Camera space using extrinsics.
        Vector3 depthCameraCoords = ConvertCameraToDepthCameraSpace(rgbCameraCoords, rgbCameraExtrinsics, depthCameraData);

        // Step 3: Map the depth camera coordinates to depth image coordinates.
        Vector2 depthPixel = MapCameraToDepthPixel(depthCameraCoords, depthImageSize, depthCameraData);

        // Step 4: Fetch the world point from the depth image based on the depth pixel.
        Vector3 worldPoint = GetWorldPointFromDepthPixel(depthPixel, depthImageWorldPoints, depthImageSize);

        return worldPoint;
    }

```

---

<div class="post-metadata">

**Author:** ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)\
**Post date:** [March 25, 2025, 3:48pm UTC](https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/5198/12 "2025-03-25T15:48:19Z")

</div>

Yes, your logic of

1. Convert YOLO pixel → RGB camera coordinates
2. Transform RGB → Depth camera coordinates
3. Depth camera coords → Depth image pixel
4. Lookup 3D from that depth pixel

is exactly the typical approach. Things that you want to consider is the resolution and FoV differences, Depth Lookup and interpolation (handle “off-pixel” coordinates), and Frame Synchronization / timestamps

---

<div class="post-metadata">

**Author:** ![yang](https://avatars.discourse-cdn.com/v4/letter/y/77aa72/32.png) [@yang](https://forum.magicleap.cloud/u/yang)\
**Post date:** [August 11, 2025, 5:19pm UTC](https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/5198/13 "2025-08-11T17:19:11Z")

</div>

Hi, as MLCamera is deprecated, is here a way to query extrinsic data with OpenXR?

---

<div class="post-metadata">

**Author:** ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)\
**Post date:** [August 11, 2025, 5:21pm UTC](https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/5198/14 "2025-08-11T17:21:39Z")

</div>

You can continue using the ML Camera API as long as you follow the steps mentioned here: [MLCamera | MagicLeap Developer Documentation](https://developer-docs.magicleap.cloud/docs/guides/unity-openxr/ml-camera/)

---

<div class="post-metadata">

**Author:** ![yang](https://avatars.discourse-cdn.com/v4/letter/y/77aa72/32.png) [@yang](https://forum.magicleap.cloud/u/yang)\
**Post date:** [December 4, 2025, 4:22pm UTC](https://forum.magicleap.cloud/t/cv-mapping-with-depth-camera/5198/15 "2025-12-04T16:22:30Z")

</div>

Thanks a lot! I followed the transformation and aligned the depth map to the RGB image. But I noticed an offset of around 10 pixels between them, is it a problem of main/tof camera calibration? The offset can be reduced if I manipulate position values in the camera poses.