# How to use depth camera DistortionCoefficients? **URL:** \ **Category:** General\ **Tags:** development, unity\ **Created:** [April 12, 2023, 2:21pm UTC](https://forum.magicleap.cloud/t/how-to-use-depth-camera-distortioncoefficients/1734 "2023-04-12T14:21:53Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![frankschoenhofer](https://avatars.discourse-cdn.com/v4/letter/f/9de0a6/32.png) [@frankschoenhofer](https://forum.magicleap.cloud/u/frankschoenhofer)\ **Post date:** [April 12, 2023, 2:21pm UTC](https://forum.magicleap.cloud/t/how-to-use-depth-camera-distortioncoefficients/1734/1 "2023-04-12T14:21:53Z")
I am trying to map depth camera data to cv camera frames to calculate depth at a given coordinate in the cv camera frames. Is there any further info or example on how to use the DistortionCoefficents of the depth camera? ---
**Author:** ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)\ **Post date:** [April 13, 2023, 4:54am UTC](https://forum.magicleap.cloud/t/how-to-use-depth-camera-distortioncoefficients/1734/2 "2023-04-13T04:54:52Z")
Hi @frankschoenhofer, happy to help woth this. What type of sample are you look for? What are you using to develop your application (Unity / Native)? ---
**Author:** ![frankschoenhofer](https://avatars.discourse-cdn.com/v4/letter/f/9de0a6/32.png) [@frankschoenhofer](https://forum.magicleap.cloud/u/frankschoenhofer)\ **Post date:** [April 13, 2023, 12:52pm UTC](https://forum.magicleap.cloud/t/how-to-use-depth-camera-distortioncoefficients/1734/3 "2023-04-13T12:52:35Z")
Hi! I'm using Unity - I am using Google Mediapipe to track human poses through the cvcamera and combine it with depth camera data to calculate joint positions in 3D. It works, but the accuracy could be better. That's why I'm looking into ways to improve the mapping between cvcamera and depth camera. Reading through the documentation I found the depth camera DistortionCoefficients, but the documentation doesn't explain much about that feature. Do you have any further readings or examples how to use the coefficients to "undistort" the image? ---
**Author:** ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)\ **Post date:** [April 14, 2023, 12:06am UTC](https://forum.magicleap.cloud/t/how-to-use-depth-camera-distortioncoefficients/1734/4 "2023-04-14T00:06:05Z")
That sounds awesome! I’d love to see it once you get it working. Regarding documentation, the distortion coefficients are not specific to Magic Leap and refer to the depth camera's projection matrix: > **[Unity - Scripting API: Camera.projectionMatrix](https://docs.unity3d.com/ScriptReference/Camera-projectionMatrix.html)** Other depth camera devices provide similar values. Since our depth camera API is experimental, it's still maturing and we are working on providing more samples in the future. For now I recommend looking at the work that has been done with existing depth cameras and applying it to our API. For Example : [AzureKinect4Unity](https://github.com/sotanmochi/AzureKinect4Unity/blob/ce69602bfc183dbb00c7d41566d3cec423f3cce2/Assets/AzureKinect4Unity/Examples/Domain/AzureKinectPointCloudVisualizer.cs) Let me know what type of example/ sample could help with this so I can share it with our voice of customers team. ---
**Author:** ![frankschoenhofer](https://avatars.discourse-cdn.com/v4/letter/f/9de0a6/32.png) [@frankschoenhofer](https://forum.magicleap.cloud/u/frankschoenhofer)\ **Post date:** [April 17, 2023, 1:21pm UTC](https://forum.magicleap.cloud/t/how-to-use-depth-camera-distortioncoefficients/1734/5 "2023-04-17T13:21:28Z")
Maybe mapping the camera images in 2d without lens distortion correction is enough. I'll try to optimize that first. The optics of the two look pretty similar. Thanks for the links - I'll look into that! ---
**Author:** ![jaime.cisneros](https://avatars.discourse-cdn.com/v4/letter/j/90db22/32.png) [@jaime.cisneros](https://forum.magicleap.cloud/u/jaime.cisneros)\ **Post date:** [April 24, 2023, 12:44am UTC](https://forum.magicleap.cloud/t/how-to-use-depth-camera-distortioncoefficients/1734/6 "2023-04-24T00:44:59Z")
I believe the depth frame is not paired with the RGB image. That means that for an RGB image taken at time t, the depth map you get does not correspond to the same t. Thus, using the 2D position of a joint from the RGB image to find the depth in the depth frame will give you inaccurate results. ---
**Author:** ![mattycorbett](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@mattycorbett](https://forum.magicleap.cloud/u/mattycorbett)\ **Post date:** [September 12, 2023, 8:32pm UTC](https://forum.magicleap.cloud/t/how-to-use-depth-camera-distortioncoefficients/1734/7 "2023-09-12T20:32:15Z")
@frankschoenhofer I am also trying to use MediaPipe on the ML2. I am having issues using a plugin written for unity here ( [GitHub - homuler/MediaPipeUnityPlugin: Unity plugin to run MediaPipe](https://github.com/homuler/MediaPipeUnityPlugin)). How did you get MediaPipe to run on the ML2? This plugin? If not, can I ask how? ---
**Author:** ![frankschoenhofer](https://avatars.discourse-cdn.com/v4/letter/f/9de0a6/32.png) [@frankschoenhofer](https://forum.magicleap.cloud/u/frankschoenhofer)\ **Post date:** [September 15, 2023, 2:44pm UTC](https://forum.magicleap.cloud/t/how-to-use-depth-camera-distortioncoefficients/1734/8 "2023-09-15T14:44:49Z")
Hi @mattycorbett - I used that same plugin. But it has to be built for Android x86\_64. I forked the repo and added the x86\_64 target to .bazelrc Was planning to contact homuler, maybe they'll integrate it. This is the fork: [GitHub - karmacod3r/MediaPipeUnityPluginAndroidX86\_64: Unity plugin to run MediaPipe graphs](https://github.com/karmacod3r/MediaPipeUnityPluginAndroidX86_64) ---
**Author:** ![mattycorbett](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@mattycorbett](https://forum.magicleap.cloud/u/mattycorbett)\ **Post date:** [September 15, 2023, 4:22pm UTC](https://forum.magicleap.cloud/t/how-to-use-depth-camera-distortioncoefficients/1734/9 "2023-09-15T16:22:25Z")
@frankschoenhofer Thank you so much for responding! This is awesome. Did you have to change the file folder structure or anything else? Just clone the repo, build for x86\_64 (did you use a specific NPK level?), and import, right? Thank you again! ---
**Author:** ![frankschoenhofer](https://avatars.discourse-cdn.com/v4/letter/f/9de0a6/32.png) [@frankschoenhofer](https://forum.magicleap.cloud/u/frankschoenhofer)\ **Post date:** [September 15, 2023, 4:44pm UTC](https://forum.magicleap.cloud/t/how-to-use-depth-camera-distortioncoefficients/1734/10 "2023-09-15T16:44:46Z")
No need to clone it. I forked it, edited and ran the github action to build it. ---
**Author:** ![frankschoenhofer](https://avatars.discourse-cdn.com/v4/letter/f/9de0a6/32.png) [@frankschoenhofer](https://forum.magicleap.cloud/u/frankschoenhofer)\ **Post date:** [September 15, 2023, 4:50pm UTC](https://forum.magicleap.cloud/t/how-to-use-depth-camera-distortioncoefficients/1734/11 "2023-09-15T16:50:10Z")
…when the action is through you can download it’s artifacts, meaning the unity package. I just merged in recent changes and started a new build I can keep you updated when and if that succeeds ---
**Author:** ![mattycorbett](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@mattycorbett](https://forum.magicleap.cloud/u/mattycorbett)\ **Post date:** [September 15, 2023, 4:52pm UTC](https://forum.magicleap.cloud/t/how-to-use-depth-camera-distortioncoefficients/1734/12 "2023-09-15T16:52:43Z")
@frankschoenhofer Thank you! ---
**Author:** ![mattycorbett](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@mattycorbett](https://forum.magicleap.cloud/u/mattycorbett)\ **Post date:** [September 15, 2023, 6:22pm UTC](https://forum.magicleap.cloud/t/how-to-use-depth-camera-distortioncoefficients/1734/13 "2023-09-15T18:22:34Z")
@frankschoenhofer The tarball works! Its now throwing the error below, but that says to me the libraries are loading! Ever had this error? ![image](https://us1.discourse-cdn.com/flex019/uploads/magicleap1/original/2X/e/e053e0d24120acd97a51eb62d84753659f563386.png) ---
**Author:** ![frankschoenhofer](https://avatars.discourse-cdn.com/v4/letter/f/9de0a6/32.png) [@frankschoenhofer](https://forum.magicleap.cloud/u/frankschoenhofer)\ **Post date:** [September 15, 2023, 6:58pm UTC](https://forum.magicleap.cloud/t/how-to-use-depth-camera-distortioncoefficients/1734/14 "2023-09-15T18:58:46Z")
No, maybe thats because mediapipe is not included in the plugin anymore. Guess mediapipe has to be built for x86 too. I used the old version 0.10.3 till now. You can download that one under „releases“ if you want to try it. Maybe I‘ll look into building mediapipe next week. Let me know if you figure that one out - must be something similar.