How to use depth camera DistortionCoefficients?

General
,
1

I am trying to map depth camera data to cv camera frames to calculate depth at a given coordinate in the cv camera frames. Is there any further info or example on how to use the DistortionCoefficents of the depth camera?

2

Hi @frankschoenhofer, happy to help woth this. What type of sample are you look for? What are you using to develop your application (Unity / Native)?

3

Hi! I'm using Unity - I am using Google Mediapipe to track human poses through the cvcamera and combine it with depth camera data to calculate joint positions in 3D. It works, but the accuracy could be better.
That's why I'm looking into ways to improve the mapping between cvcamera and depth camera. Reading through the documentation I found the depth camera DistortionCoefficients, but the documentation doesn't explain much about that feature. Do you have any further readings or examples how to use the coefficients to "undistort" the image?

1 Like
4

That sounds awesome! I’d love to see it once you get it working.

Regarding documentation, the distortion coefficients are not specific to Magic Leap and refer to the depth camera's projection matrix:

Other depth camera devices provide similar values. Since our depth camera API is experimental, it's still maturing and we are working on providing more samples in the future. For now I recommend looking at the work that has been done with existing depth cameras and applying it to our API.

For Example : AzureKinect4Unity

Let me know what type of example/ sample could help with this so I can share it with our voice of customers team.

5

Maybe mapping the camera images in 2d without lens distortion correction is enough. I'll try to optimize that first. The optics of the two look pretty similar.

Thanks for the links - I'll look into that!

6

I believe the depth frame is not paired with the RGB image. That means that for an RGB image taken at time t, the depth map you get does not correspond to the same t. Thus, using the 2D position of a joint from the RGB image to find the depth in the depth frame will give you inaccurate results.