# Depth Data Quality

**URL:** https://forum.magicleap.cloud/t/depth-data-quality/3926
**Category:** General
**Tags:** feature-request, pixel-sensor, feedback, depth-camera
**Created:** [August 8, 2024, 3:11am UTC](https://forum.magicleap.cloud/t/depth-data-quality/3926 "2024-08-08T03:11:00Z")
**Posts on this page:** 11
**Page:** 1

<div class="post-metadata">

### Author: ![GspLi](https://avatars.discourse-cdn.com/v4/letter/g/35a633/32.png) [@GspLi](https://forum.magicleap.cloud/u/GspLi)
#### Post date: [August 8, 2024, 3:11am UTC](https://forum.magicleap.cloud/t/depth-data-quality/3926/1 "2024-08-08T03:11:00Z")

</div>

I was investigating the depth quality between HoloLens2 and Magic Leap2. I took Long-depth FPS1 images from both devices. What I found out:

- After PCD conversion, points arrangement looks completely different (Image).
- PCD captured from HL2 looks smoother compared to ML2
- Captured PCD depth is not correct for both devices. HL2 is way worse than ML2. For ML2, it aligns alright but for HL2, it is like 5-6cm below the surface level.

I don't have many clues on the results. If anyone worked on PCD capture using MagicLeap.OpenXR.Features.PixelSensors, would you please share your thoughts?

![image](https://us1.discourse-cdn.com/flex019/uploads/magicleap1/original/2X/6/6b120903e58a9daed9def4280596f033dac06a10.png)

---

<div class="post-metadata">

### Author: ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)
#### Post date: [August 8, 2024, 4:09pm UTC](https://forum.magicleap.cloud/t/depth-data-quality/3926/2 "2024-08-08T16:09:45Z")

</div>



---

<div class="post-metadata">

### Author: ![GspLi](https://avatars.discourse-cdn.com/v4/letter/g/35a633/32.png) [@GspLi](https://forum.magicleap.cloud/u/GspLi)
#### Post date: [August 9, 2024, 9:07am UTC](https://forum.magicleap.cloud/t/depth-data-quality/3926/4 "2024-08-09T09:07:53Z")

</div>

Here is the code that I have used to generate the PCD:

```auto
private void ProcessAndCleanData(in PixelSensorFrame frame, in PixelSensorDepthConfidenceBuffer conf_buffer, in PixelSensorDepthFlagBuffer flag_buffer, in Pose sensorPose, in PixelSensorPinholeIntrinsics pinholeIntrinsics)
        {
            uint height, width;
            byte[] depthData = ProcessFrame(in frame, out height, out width);
            // byte[] confidenceData = ProcessDepthConfidenceData(in conf_buffer);
            byte[] flagData = ProcessDepthFlagData(in flag_buffer);

            if (depthData.Length == 0 || flagData.Length == 0)
            {
                progressHandler.Fail("No data to process.");
                return;
            }

            List<Vector3> points = new();

            for (int y = 0; y < height; y++)
            {
                for (int x = 0; x < width; x++)
                {
                    var i = y * width + x;

                    float depth = BitConverter.ToSingle(depthData, (int)(i * sizeof(float)));
                    UInt32 pixelFlag = BitConverter.ToUInt32(flagData, (int)(i * sizeof(UInt32)));

                    // TODO: PixelSensorDepthFlags works but not as expected in the API documentation (not sure if it's a bug)
                    // Flag & Valid is expected to be 1 but it's 0
                    if (!DepthDataCleaning || (pixelFlag & (UInt32)PixelSensorDepthFlags.Valid) == 0)
                    {
                        Vector2 focalLength = pinholeIntrinsics.FocalLength;
                        Vector2 pp = pinholeIntrinsics.PrincipalPoint;
                        (var xNorm, var yNorm) = DepthUndistortion ? Undistort(x, y, pinholeIntrinsics) : ((x - pp.x) / focalLength.x, (y - pp.y) / focalLength.y);

                        var zNorm = depth / Mathf.Sqrt(xNorm * xNorm + yNorm * yNorm + 1);

                        var xConverted = xNorm * zNorm;
                        var yConverted = yNorm * zNorm;

                        var point = new Vector3(xConverted, yConverted, zNorm);

                        points.Add(point);
                    }

                }
            }

            if (WriteToFile)
            {
                string formattedTime = DateTimeOffset.FromUnixTimeMilliseconds(frame.CaptureTime / 1000).ToString(@"yyyy-MM-dd hh\:mm\:ss");
                string filePath = Application.persistentDataPath + $"/DEPTH {formattedTime}-PCD-" + (DepthUndistortion ? "Undistort" : "Raw") + ".pcd";
                _ = WritePCDAsync(points, filePath);
            }

            pointCloudRenderer.Render(points.ToArray(), DepthUndistortion ? Color.cyan : Color.green);
            pointCloudRenderer.transform.SetPositionAndRotation(sensorPose.position, sensorPose.rotation);

        }

        private (float x, float y) Undistort(int x, int y, PixelSensorPinholeIntrinsics pinholeIntrinsics)
        {
            // Pinhole Distortion Model
            // [k1, k2, p1, p2, k3]
            var K1 = (float)pinholeIntrinsics.Distortion[0];
            var K2 = (float)pinholeIntrinsics.Distortion[1];
            var P1 = (float)pinholeIntrinsics.Distortion[2];
            var P2 = (float)pinholeIntrinsics.Distortion[3];
            var K3 = (float)pinholeIntrinsics.Distortion[4];
            var cx = pinholeIntrinsics.PrincipalPoint.x;
            var cy = pinholeIntrinsics.PrincipalPoint.y;
            var fx = pinholeIntrinsics.FocalLength.x;
            var fy = pinholeIntrinsics.FocalLength.y;

            var x_ud = (x - cx) / fx;
            var y_ud = (y - cy) / fy;

            var r2 = x_ud * x_ud + y_ud * y_ud;
            var r4 = r2 * r2;
            var r6 = r4 * r2;

            var xr = x_ud * (1 + K1 * r2 + K2 * r4 + K3 * r6);
            var yr = y_ud * (1 + K1 * r2 + K2 * r4 + K3 * r6);

            var xt = 2 * P1 * x_ud * y_ud + P2 * (r2 + 2 * x_ud * x_ud);
            var yt = P1 * (r2 + 2 * y_ud * y_ud) + 2 * P2 * x_ud * y_ud;

            return (xr + xt, yr + yt);
        }

        private byte[] ProcessFrame(in PixelSensorFrame frame, out uint height, out uint width)
        {
            height = 0;
            width = 0;

            if (!frame.IsValid || frame.Planes.Length == 0)
            {
                return new byte[] { };
            }
            string formattedTime = DateTimeOffset.FromUnixTimeMilliseconds(frame.CaptureTime / 1000).ToString(@"yyyy-MM-dd hh\:mm\:ss");

            var planeData = frame.Planes[0].ByteData;
            byte[] planeBytes = planeData.ToArray();

            if (WriteToFile)
                File.WriteAllBytes(Application.persistentDataPath + $"/DEPTH {formattedTime}-IMG.bin", planeBytes);
            progressHandler.Log("Frame processing finished.");

            height = frame.Planes[0].Height;
            width = frame.Planes[0].Width;

            return planeBytes;
        }

```

---

<div class="post-metadata">

### Author: ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)
#### Post date: [August 10, 2024, 12:51am UTC](https://forum.magicleap.cloud/t/depth-data-quality/3926/5 "2024-08-10T00:51:42Z")

</div>

Thank you for the post. Do you mind providing us with a little extra information? It looks like you are seeing noise in the Magic Leap 2 Depth sensor data. Would it be possible for you to provide images of the surface that you are testing the devices on? What depth configuration are you using when activating the pixel sensor?

---

<div class="post-metadata">

### Author: ![GspLi](https://avatars.discourse-cdn.com/v4/letter/g/35a633/32.png) [@GspLi](https://forum.magicleap.cloud/u/GspLi)
#### Post date: [August 11, 2024, 7:18am UTC](https://forum.magicleap.cloud/t/depth-data-quality/3926/6 "2024-08-11T07:18:12Z")

</div>

![img](https://us1.discourse-cdn.com/flex019/uploads/magicleap1/original/2X/6/68159da5d95638b2fd1852e31cd43f49ac225966.jpeg)

I was testing on multiple surfaces. This is the image of the surface that I used to generate the earlier PCD. Generally, I found

- ML-captured PCDs are dense but have much noise compared to HL.
- I checked it with the old MLSDK and the new OpenXR, and the results are the same.
- I could not find any differences between Distorted and Undistorted PCD using the new OpenXR API. But there were differences in the old version.
- FOV is not adjustable which also causes a lot of noise issues.

Here is the snippet of the code:

```auto
        public uint targetStream
        {
            get { return DepthRange > 1.0f ? (uint)0 : (uint)1; }
        }

        public enum LongRangeUpdateRate
        {
            OneFps = 1, FiveFps = 5
        }
        public enum ShortRangeUpdateRate
        {
            FiveFps = 5, ThirtyFps = 30, SixtyFps = 60
        }

        [Header("ShortRange =< 1m")] private ShortRangeUpdateRate SRUpdateRate = ShortRangeUpdateRate.FiveFps;
        [Header("LongRange > 1m")] private LongRangeUpdateRate LRUpdateRate = LongRangeUpdateRate.FiveFps;

        private MagicLeapPixelSensorFeature pixelSensorFeature;

        private void InitializePixelSensorFeature()
        {
            // Get the Magic Leap Pixel Sensor Feature from the OpenXR settings
            pixelSensorFeature = OpenXRSettings.Instance.GetFeature<MagicLeapPixelSensorFeature>();
            if (pixelSensorFeature == null || !pixelSensorFeature.enabled)
            {
                progressHandler.Fail("Pixel Sensor Feature Not Found or Not Enabled!");
                return;
            }
            // Permissions.RequestPermission(UnityEngine.Android.Permission.Camera);
            Permissions.RequestPermission(Permissions.DepthCamera,
            (permission) =>
            {
                if (permission.Contains(Permissions.DepthCamera))
                    depthSensorId = FindAndInitializeSensor("/pixelsensor/depth/center");
            },
            (permission) =>
            {
                progressHandler.Fail("Permission Denied: " + permission);
            });

            progressHandler.Log("InitializePixelSensorFeature finished");
        }
        private PixelSensorId FindAndInitializeSensor(string pixelSensorPath)
        {
            var sensors = pixelSensorFeature.GetSupportedSensors();
            PixelSensorId sensorId = sensors.Find(x => x.XrPathString == pixelSensorPath);

            if (pixelSensorFeature.CreatePixelSensor(sensorId))
            {
                progressHandler.Log("Sensor created successfully " + pixelSensorPath);
            }
            else
            {
                progressHandler.Fail("Failed to create sensor " + pixelSensorPath);
                throw new Exception("Failed to create sensor " + pixelSensorPath);
            }

            StartCoroutine(StartSensorStream(sensorId));
            return sensorId;
        }

        private PixelSensorCapabilityType[] _targetCapabilityTypes = new[]
        {
            PixelSensorCapabilityType.UpdateRate,
            PixelSensorCapabilityType.Format,
            PixelSensorCapabilityType.Resolution,
            PixelSensorCapabilityType.Depth,
        };

        // Coroutine to configure stream and start sensor streams
        // targetStream: 0 for Long, 1 for short
        private IEnumerator StartSensorStream(PixelSensorId sensorId)
        {
            pixelSensorFeature.GetPixelSensorCapabilities(sensorId, targetStream, out var capabilities);
            foreach (var pixelSensorCapability in capabilities)
            {
                if (!_targetCapabilityTypes.Contains(pixelSensorCapability.CapabilityType))
                {
                    continue;
                }

                // More details about the capability
                if (pixelSensorFeature.QueryPixelSensorCapability(sensorId, pixelSensorCapability.CapabilityType, targetStream, out PixelSensorCapabilityRange range) && range.IsValid)
                {
                    if (range.CapabilityType == PixelSensorCapabilityType.UpdateRate)
                    {
                        var configData = new PixelSensorConfigData(range.CapabilityType, targetStream);
                        configData.IntValue = DepthRange > 1 ? (uint)LRUpdateRate : (uint)SRUpdateRate;
                        pixelSensorFeature.ApplySensorConfig(sensorId, configData);
                    }
                    else if (range.CapabilityType == PixelSensorCapabilityType.Format)
                    {
                        var configData = new PixelSensorConfigData(range.CapabilityType, targetStream);
                        configData.IntValue = (uint)range.FrameFormats[UseRawDepth ? 1 : 0];
                        pixelSensorFeature.ApplySensorConfig(sensorId, configData);
                    }
                    else if (range.CapabilityType == PixelSensorCapabilityType.Resolution)
                    {
                        var configData = new PixelSensorConfigData(range.CapabilityType, targetStream);
                        configData.VectorValue = range.ExtentValues[0];
                        pixelSensorFeature.ApplySensorConfig(sensorId, configData);
                    }
                    else if (range.CapabilityType == PixelSensorCapabilityType.Depth)
                    {
                        var configData = new PixelSensorConfigData(range.CapabilityType, targetStream);
                        configData.FloatValue = DepthRange;
                        pixelSensorFeature.ApplySensorConfig(sensorId, configData);
                    }
                }
            }

            var configureOperation = pixelSensorFeature.ConfigureSensor(sensorId, new[] { targetStream });

            yield return configureOperation;

            if (!configureOperation.DidOperationSucceed)
            {
                progressHandler.Fail("Failed to configure sensor.");
            }

            progressHandler.Log("Sensor configuration finished.");

            // Obtain the supported metadata types from the sensor
            Dictionary<uint, PixelSensorMetaDataType[]> supportedMetadataTypes = new();

            if (pixelSensorFeature.EnumeratePixelSensorMetaDataTypes(sensorId, targetStream, out var metaDataTypes))
            {
                supportedMetadataTypes[targetStream] = metaDataTypes;
            }

            // Start the sensor with the default configuration and specify that all of the meta data should be requested.
            PixelSensorAsyncOperationResult sensorStartAsyncResult =
                pixelSensorFeature.StartSensor(sensorId, new[] { targetStream }, supportedMetadataTypes);

            yield return sensorStartAsyncResult;

            if (!sensorStartAsyncResult.DidOperationSucceed)
            {
                progressHandler.Fail("Failed to start sensor.");
            }

            progressHandler.Log("Sensor start finished. Monitoring data...");
        }

```

---

<div class="post-metadata">

### Author: ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)
#### Post date: [August 11, 2024, 9:47pm UTC](https://forum.magicleap.cloud/t/depth-data-quality/3926/7 "2024-08-11T21:47:50Z")

</div>

Thank you for that information, I have shared your feedback with our voice of customer team.

Note our depth sensor API provides the data without additional filtering. Glossy or dark surfaces will also affect the data since those materials interact with the light emitted by the sensor.

---

<div class="post-metadata">

### Author: ![GspLi](https://avatars.discourse-cdn.com/v4/letter/g/35a633/32.png) [@GspLi](https://forum.magicleap.cloud/u/GspLi)
#### Post date: [August 12, 2024, 6:05am UTC](https://forum.magicleap.cloud/t/depth-data-quality/3926/8 "2024-08-12T06:05:23Z")

</div>

It would be really helpful if the Magic Leap team could provide configuration guidelines and code for Depth to PCD.

---

<div class="post-metadata">

### Author: ![GspLi](https://avatars.discourse-cdn.com/v4/letter/g/35a633/32.png) [@GspLi](https://forum.magicleap.cloud/u/GspLi)
#### Post date: [August 20, 2024, 5:06pm UTC](https://forum.magicleap.cloud/t/depth-data-quality/3926/9 "2024-08-20T17:06:57Z")

</div>

Hi @kbabilinski, any update on this?

---

<div class="post-metadata">

### Author: ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)
#### Post date: [August 20, 2024, 7:00pm UTC](https://forum.magicleap.cloud/t/depth-data-quality/3926/10 "2024-08-20T19:00:38Z")

</div>

I have requested the feedback on your behalf which has now been marked as a feature request. If you wish to use the filtered depth data, you can use the Meshing feature instead that tries to reduce the noise.

---

<div class="post-metadata">

### Author: ![GspLi](https://avatars.discourse-cdn.com/v4/letter/g/35a633/32.png) [@GspLi](https://forum.magicleap.cloud/u/GspLi)
#### Post date: [September 30, 2024, 5:58am UTC](https://forum.magicleap.cloud/t/depth-data-quality/3926/11 "2024-09-30T05:58:12Z")

</div>

Hi @kbabilinski,

Any updates?

---

<div class="post-metadata">

### Author: ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)
#### Post date: [September 30, 2024, 4:21pm UTC](https://forum.magicleap.cloud/t/depth-data-quality/3926/12 "2024-09-30T16:21:48Z")

</div>

I do not have any updates regarding your feature request at this time. You may want to use the Meshing feature instead of the raw depth for development.