CV mapping with Depth Camera

Unity Development
,
1

Unity Editor version: 2022.3.21f1
ML2 OS version: 1.3.1
Unity SDK version: 1.5.0
Host OS: Windows 11

Hello,
I am working on a project where I detect interests points with the CV camera and I want to obtain the depth of those points thanks to the depth camera (in order to reproject them precisely in 3D).
I tested the DepthCamera example scene from Uniy Example package (1.12) but I have several questions.

  • Can we obtain values in meters directly from the MLDepthCamera.CaptureFlags.DepthImage ? ''Depth map stores the depth data from the depth camera. Depth is represented in meters and gives the radial distance of the real world location from the depth camera coordinate frame. '' (documentation)

  • According to some previous topics you need to first undistort the depth image and to map it to the CV camera image. Is there any function or code snippets that would be helpful ?
    I suppose that with distortion, different resolution and fps the mapping can be challenging.

2

Since the depth sensor api is experimental, the documentation is limited due to the stability of the API. That said , here is a related forum post the shows how to render the depth point cloud. If you cant see the texture you may need to make sure that “reset head pose” on the magic leap camera component is disabled: Processing the depth frames - #13 by kbabilinski

3

Thank you very much for your answer I managed to display some points from the depth camera in real world. But do you know if there is a function such as the one discussed last year on this topic MLVec2f RgbPosToDepthPos(MLVec2f rgbPixelCoords, MLVec2f rgbImageResolution); ?

It would be very helpful to get the precise point correspondance between the CV camera and the depth camera. So I can retrieve only the points I need from the depth camera.

4

We do not have this type of API at this time, but I can share your feedback with our voice of customer team.

One thing that you need to keep in mind when trying to cast a ray from the CV camera and get a hit from the depth image is that the CV camera and depth camera are not in the same place and do not have the same intrinsics or distortion coefficients.

There are a couple of different ways one could approach this - for example, the azure kinect api has a set of transform function that perform mesh reprojection of the images (Use Azure Kinect Sensor SDK image transformations | Microsoft Learn). Once you are able to transform the contents of one image to match the properties of the other then you can just query the depth image at the same pixel location as the color image to get your depth value.

5

Thank you for the reply, as a follow up question is the offset between the depth and CV cameras available in the API?

I created and tested the following pipeline with a made-up cameras offset and obtained this Error: MLCVCameraGetFramePose in the Magic Leap API failed. Reason : MLResult_PoseNotFound which is according to documentation : Coordinate Frame is valid, but not found in the current pose snapshot

  1. I thought about transforming the normalized pixel coordinates from my detection model to normalized CV cam coordinates :

ndc_coordinates = ((undistortedCoordinates * new Vector2(resolution.x, resolution.y)) - resultExtras.Intrinsics.Value.PrincipalPoint) / resultExtras.Intrinsics.Value.FocalLength;

  1. Then to the normalized depth cam coordinates system :
    Made-up for now, but is there an API function or do I need to calculate the offset from each cameras position ?

  2. Afterwards, I retrieve the point depth from a dictionary that I created during depth mapping.

  3. And finally convert to world system to display the point in the headset:

Matrix4x4 cameraToWorldMatrix = depthPointCloudTest.cameraToWorldMatrix;
Vector3 worldPoint = cameraToWorldMatrix.MultiplyPoint3x4(cameraPoint);

It is the same error as this topic but i believe it is due to my wrong offset in coordinates calcul. Or maybe the dictionary that i created during the depth map is not really simultaneous to the detection from CV camera.

6

Both RGB and Depth camera provide their pose value with every frame they capture. These values should be queried each frame because the capture is async, so their relative positions can change depending on when each frame was captured.

Regarding the MLCVCameraGetFramePose, this error suggests that too much time has passed since the frame was captured to get it's pose. This can occur if the application is experiences a frame drop.

The CV Camera pose should be obtained as soon as the camera callback is triggered and then cached if you need to process the image at a later time.

7

Thanks for these informations

To give more insight on the MLCVCameraGetFramePose error :
I have two public methods in CameraCaptureVisualizer script, one to retrieve the texture on which is then applied my detection algorithm. And also one to retrieve the extras of the camera to get the camera pose and its intrinsics.

public Texture GetTexture()
        {
            UnityEngine.Debug.Log("send texture" + mTexture.ToString());
            return mTexture;
        }

public MLCamera.ResultExtras GetPose()
        {
            UnityEngine.Debug.Log("send CV intrinsics + pose" + extras.ToString());
            return extras;
        }

The error seem to happen when I try accessing the ResultExtras after my detection script :

detecter.ProcessImage(cameraCaptureVisualizer.GetTexture(), BlazePoseModel.lite, humanExistThreshold);
MLCamera.ResultExtras resultExtras = cameraCaptureVisualizer.GetPose();
Vector2Int resolution = new Vector2Int((int)resultExtras.Intrinsics.Value.Width, (int)resultExtras.Intrinsics.Value.Height);

How should I proceed since I only need the CV pose of the frame treated by my detection ?

And since I also need the depth mapping/depth camera pose of this same frame, how to be sure to query the correct mapping/depth camera pose? Because for now I have the separate script that get the depth point cloud but there is no frame matching.

8

I recommend obtaining the pose of the image before processing it by your detection algorithm. This will ensure that any potential slow down will not affect the ability to get that frame's pose. That said, you may want to see if you can offload the detection process to another thread so that the application can hit the 60fps requirement every frame.

Regarding synchronization, the Magic Leap 2 does not provide synchronized frames, so you will need to synchronize the based on the frame time stamps inside your application. You can convert the timestamp using the following API so by using

var result = MLTime.ConvertMLTimeToSystemTime(frame.TimeStamp, out long timestampNs);

That said, I have reported your request to our voice of customer team.