CV mapping with Depth Camera

[psyje6]

Unity Editor version : 2022.3.42f1
ML2 OS version : 1.8.0
Unity SDK version : 2.5.0
Host OS : MacOS

Hi, I am working on a project where I would like to use the 2D coordinates from my object detection model and convert them to world space using the depth camera. But I’m a bit confused on where to start, as the other posts here seem to be using a different API then is in the current documentation. Could you please point me in the right direction. Thank you!

Example:

Depth Camera Example | MagicLeap Developer Documentation

CV mapping with Depth Camera

We do not have this type of API at this time, but I can share your feedback with our voice of customer team. One thing that you need to keep in mind when trying to cast a ray from the CV camera and get a hit from the depth image is that the CV camera and depth camera are not in the same place and do not have the same intrinsics or distortion coefficients. There are a couple of different ways one could approach this - for example, the azure kinect api has a set of transform function that perfor…

[kbabilinski]

I think the other link was in reference to how another device aligns the data from the two sensors, not that it is specific to the device. You are correct to use the Pixel Sensor API.

We do not provide SDK functionality to align or synchronize sensors, so this would need to be done using custom logic in your application. Note that the RGB camera and the Depth camera are not located at the same place on the Magic Leap Headset. So you will need to use the Intrinsic and Extrinsic data to align the RGB and Depth images.

[psyje6]

Thank you! That helped me make a good start. However I’m getting stuck when getting the pose of the frame. I’m doing it as soon as the camera callback is triggered as suggested in the other post. But I’m getting the same MLCVCameraGetFramePose error and I’m really confused as to why. Here is my function:

private void OnCaptureRawVideoFrameAvailable(MLCameraBase.CameraOutput cameraOutput,
            MLCameraBase.ResultExtras resultExtras,
            MLCameraBase.Metadata metadata)
        {

            MLResult result = MLCVCamera.GetFramePose(resultExtras.VCamTimestamp, out Matrix4x4 outMatrix);
            if (result.IsOk)
            {
                string cameraExtrinsics = "Camera Extrinsics";
                cameraExtrinsics += "Position " + outMatrix.GetPosition();
                cameraExtrinsics += "Rotation " + outMatrix.rotation;

                CameraExtrinsics.position = outMatrix.GetPosition();
                CameraExtrinsics.rotation = Matrix4x4.Rotate(outMatrix.rotation);
                CameraExtrinsics.timestamp = resultExtras.VCamTimestamp;

                Debug.Log(cameraExtrinsics);
            }

            CameraIntrinsics.fx = resultExtras.Intrinsics.Value.FocalLength.x;
            CameraIntrinsics.fy = resultExtras.Intrinsics.Value.FocalLength.y;

            CameraIntrinsics.cx = resultExtras.Intrinsics.Value.PrincipalPoint.x;
            CameraIntrinsics.cy = resultExtras.Intrinsics.Value.PrincipalPoint.y;

            UpdateRGBTexture(ref videoTexture, cameraOutput.Planes[0]);
}

[kbabilinski]

If you are using OpenXR , you will need to make sure the Perception Snapshots is enabled in the OpenXR Magic Leap Support Feature in your project settings

MLCamera | MagicLeap Developer Documentation

[psyje6]

I had enabled it before but am still getting the same error.

[psyje6]

This is the specific error I am getting

leapcore/frameworks/perception/data_sources/include/pad/xpad_data_source.h(105) GetClosestTimestampedData():
ERR: Data Not Found for timestamp: 1022446282us, now time: 1045159819us
Error Unity Error: MLCVCameraGetFramePose in the Magic Leap API failed. Reason: MLResult_PoseNotFound 
Error Unity UnityEngine.XR.MagicLeap.MLResult:DidNativeCallSucceed(Code, String, Predicate`1, Boolean)
Error Unity UnityEngine.XR.MagicLeap.MLCVCamera:InternalGetFramePose(CameraID, MLTime, Matrix4x4&)
Error Unity YoloHolo.Services.MLImageAcquiringService:OnCaptureRawVideoFrameAvailable(CameraOutput, ResultExtras, Metadata)
Error Unity UnityEngine.XR.MagicLeap.Native.DispatchPayload3`3:Dispatch()
Error Unity UnityEngine.XR.MagicLeap.Native.MLThreadDispatch:DispatchAll()

[kbabilinski]

You will need to make sure to get the pose of the camera within 500ms of the callback. This means that you need to maintain a high frame rate, otherwise the timestamp might become invalid.

---

You can also use the OpenXR Pixel Sensor logic to obtain the camera pose. This way you don’t have to convert between OpenXR and MLSDK poses. Note, that if you are using the MLCamera , you can just use the following example without additional configuration. Just call GetSensorPose in the callback.

API Overview | MagicLeap Developer Documentation

Callback example

    private MagicLeapPixelSensorFeature _pixelSensorFeature;
    private PixelSensorId _sensorType;
    private XROrigin _xrOrigin;
...

    void RawVideoFrameAvailable(MLCamera.CameraOutput output, MLCamera.ResultExtras extras, MLCameraBase.Metadata metadataHandle)
    {
        if (output.Format == MLCamera.OutputFormat.RGBA_8888)
        {
            //Flips the frame vertically so it does not appear upside down.
            MLCamera.FlipFrameVertically(ref output);
            UpdateRGBTexture(ref _videoTextureRgb, output.Planes[0], _screenRendererRGB);
        }

        if(_pixelSensorFeature == null)
        {
            _pixelSensorFeature = OpenXRSettings.Instance.GetFeature<MagicLeapPixelSensorFeature>();
            _sensorType = _pixelSensorFeature.GetSupportedSensors().Find(x=> x.SensorName == "Picture Center"); // Simplified for example

            bool wasCreated = _pixelSensorFeature.CreatePixelSensor(_sensorType);
        }


        Pose sensorPose = _pixelSensorFeature.GetSensorPose(_sensorType);

        // Updates the Sensor Pose to be relative to the XR Origin
        if (_xrOrigin = FindAnyObjectByType<XROrigin>())
        {
            Vector3 worldPosition = _xrOrigin.CameraFloorOffsetObject.transform.TransformPoint(sensorPose.position);
            Quaternion worldRotation = _xrOrigin.transform.rotation * sensorPose.rotation;
            // Update the existing pose
            sensorPose = new Pose(worldPosition, worldRotation);
        }

        Debug.Log("Sensor Position:" + sensorPose.position);
        Debug.Log("Sensor Rotation:" + sensorPose.rotation);

    }

[psyje6]

@kbabilinski

f (_xrOrigin = FindAnyObjectByType<XROrigin>())
        {
            Vector3 worldPosition = _xrOrigin.CameraFloorOffsetObject.transform.TransformPoint(sensorPose.position);
            Quaternion worldRotation = _xrOrigin.transform.rotation * sensorPose.rotation;
            // Update the existing pose
            sensorPose = new Pose(worldPosition, worldRotation);
        }

Hi thank you for this. What does it mean that the sensor pose is relative to the XR origin?
I am using the MLDepthCamera like this, I’m assuming there’s no difference in what the PixelSensor and the MLDepthCamera return?:

public override void Update()
        {
            //Debug.Log("Depth camera update");
            if (!permissionGranted || !MLDepthCamera.IsConnected) return;

            var result = MLDepthCamera.GetLatestDepthData(0, out MLDepthCamera.Data data);
            isFrameAvailable = result.IsOk;
            if (isFrameAvailable)
            {
                lastData = data;

                DepthCameraIntrinsics.fx = data.Intrinsics.FocalLength.x;
                DepthCameraIntrinsics.fy = data.Intrinsics.FocalLength.y;
                DepthCameraIntrinsics.cx = data.Intrinsics.PrincipalPoint.x;
                DepthCameraIntrinsics.cy = data.Intrinsics.PrincipalPoint.y;
                DepthCameraIntrinsics.width = data.Intrinsics.Width;
                DepthCameraIntrinsics.height = data.Intrinsics.Height;

                DepthCameraExtrinsics.position = data.Position;
                DepthCameraExtrinsics.rotation = Matrix4x4.Rotate(data.Rotation);
                DepthCameraExtrinsics.timestamp = data.FrameTimestamp;


            }
        }

Also do you have any advice on aligning the image from the CV Camera and depth camera as I am struggling to.

[psyje6]

I was also wondering if you could explain how to access the depth information from the MLDepthCamera.FrameBuffer depthFrame. I’m currently converting it into an array to get the depth value at each pixel, but my depth values seem to be too high.

[kbabilinski]

Regarding the Pose:

When using the MLSDK APIs with OpenXR , you will need to set the App into Unbounded tracking space. As mentioned here : MLCamera | MagicLeap Developer Documentation

This is because the MLSDK returns a Pose in a diferent space which is the equivalent of OpenXR’s unbounded.

---

Regarding the Depth Camera API:

Please note, the MLSDK API (MLDepthCamera) has been deprecated in favor of the OpenXR Magic Leap Pixel Sensor API. This means that the API will not receive updates and is no longer supported. That said, here is an old post regarding the ML Depth Camera:

Processing the depth frames

Here is a script that sends an array of world points to a "point cloud renderer" . I have not included the point cloud renderer code but the script should be enough to demonstrate how to obtain the undistorted depth data. A few things to note. You may want to check if the depth point falls into the user's field of view and exclude any depth points that are not visible. This will ensure that the depth image aligns to the world more closely. Although, the point cloud is undistorted, some skewing …