GspLi
August 8, 2024, 3:11am
1
I was investigating the depth quality between HoloLens2 and Magic Leap2. I took Long-depth FPS1 images from both devices. What I found out:
After PCD conversion, points arrangement looks completely different (Image).
PCD captured from HL2 looks smoother compared to ML2
Captured PCD depth is not correct for both devices. HL2 is way worse than ML2. For ML2, it aligns alright but for HL2, it is like 5-6cm below the surface level.
I don't have many clues on the results. If anyone worked on PCD capture using MagicLeap.OpenXR.Features.PixelSensors, would you please share your thoughts?
GspLi
August 9, 2024, 9:07am
4
Here is the code that I have used to generate the PCD:
private void ProcessAndCleanData(in PixelSensorFrame frame, in PixelSensorDepthConfidenceBuffer conf_buffer, in PixelSensorDepthFlagBuffer flag_buffer, in Pose sensorPose, in PixelSensorPinholeIntrinsics pinholeIntrinsics)
{
uint height, width;
byte[] depthData = ProcessFrame(in frame, out height, out width);
// byte[] confidenceData = ProcessDepthConfidenceData(in conf_buffer);
byte[] flagData = ProcessDepthFlagData(in flag_buffer);
if (depthData.Length == 0 || flagData.Length == 0)
{
progressHandler.Fail("No data to process.");
return;
}
List<Vector3> points = new();
for (int y = 0; y < height; y++)
{
for (int x = 0; x < width; x++)
{
var i = y * width + x;
float depth = BitConverter.ToSingle(depthData, (int)(i * sizeof(float)));
UInt32 pixelFlag = BitConverter.ToUInt32(flagData, (int)(i * sizeof(UInt32)));
// TODO: PixelSensorDepthFlags works but not as expected in the API documentation (not sure if it's a bug)
// Flag & Valid is expected to be 1 but it's 0
if (!DepthDataCleaning || (pixelFlag & (UInt32)PixelSensorDepthFlags.Valid) == 0)
{
Vector2 focalLength = pinholeIntrinsics.FocalLength;
Vector2 pp = pinholeIntrinsics.PrincipalPoint;
(var xNorm, var yNorm) = DepthUndistortion ? Undistort(x, y, pinholeIntrinsics) : ((x - pp.x) / focalLength.x, (y - pp.y) / focalLength.y);
var zNorm = depth / Mathf.Sqrt(xNorm * xNorm + yNorm * yNorm + 1);
var xConverted = xNorm * zNorm;
var yConverted = yNorm * zNorm;
var point = new Vector3(xConverted, yConverted, zNorm);
points.Add(point);
}
}
}
if (WriteToFile)
{
string formattedTime = DateTimeOffset.FromUnixTimeMilliseconds(frame.CaptureTime / 1000).ToString(@"yyyy-MM-dd hh\:mm\:ss");
string filePath = Application.persistentDataPath + $"/DEPTH {formattedTime}-PCD-" + (DepthUndistortion ? "Undistort" : "Raw") + ".pcd";
_ = WritePCDAsync(points, filePath);
}
pointCloudRenderer.Render(points.ToArray(), DepthUndistortion ? Color.cyan : Color.green);
pointCloudRenderer.transform.SetPositionAndRotation(sensorPose.position, sensorPose.rotation);
}
private (float x, float y) Undistort(int x, int y, PixelSensorPinholeIntrinsics pinholeIntrinsics)
{
// Pinhole Distortion Model
// [k1, k2, p1, p2, k3]
var K1 = (float)pinholeIntrinsics.Distortion[0];
var K2 = (float)pinholeIntrinsics.Distortion[1];
var P1 = (float)pinholeIntrinsics.Distortion[2];
var P2 = (float)pinholeIntrinsics.Distortion[3];
var K3 = (float)pinholeIntrinsics.Distortion[4];
var cx = pinholeIntrinsics.PrincipalPoint.x;
var cy = pinholeIntrinsics.PrincipalPoint.y;
var fx = pinholeIntrinsics.FocalLength.x;
var fy = pinholeIntrinsics.FocalLength.y;
var x_ud = (x - cx) / fx;
var y_ud = (y - cy) / fy;
var r2 = x_ud * x_ud + y_ud * y_ud;
var r4 = r2 * r2;
var r6 = r4 * r2;
var xr = x_ud * (1 + K1 * r2 + K2 * r4 + K3 * r6);
var yr = y_ud * (1 + K1 * r2 + K2 * r4 + K3 * r6);
var xt = 2 * P1 * x_ud * y_ud + P2 * (r2 + 2 * x_ud * x_ud);
var yt = P1 * (r2 + 2 * y_ud * y_ud) + 2 * P2 * x_ud * y_ud;
return (xr + xt, yr + yt);
}
private byte[] ProcessFrame(in PixelSensorFrame frame, out uint height, out uint width)
{
height = 0;
width = 0;
if (!frame.IsValid || frame.Planes.Length == 0)
{
return new byte[] { };
}
string formattedTime = DateTimeOffset.FromUnixTimeMilliseconds(frame.CaptureTime / 1000).ToString(@"yyyy-MM-dd hh\:mm\:ss");
var planeData = frame.Planes[0].ByteData;
byte[] planeBytes = planeData.ToArray();
if (WriteToFile)
File.WriteAllBytes(Application.persistentDataPath + $"/DEPTH {formattedTime}-IMG.bin", planeBytes);
progressHandler.Log("Frame processing finished.");
height = frame.Planes[0].Height;
width = frame.Planes[0].Width;
return planeBytes;
}
Thank you for the post. Do you mind providing us with a little extra information? It looks like you are seeing noise in the Magic Leap 2 Depth sensor data. Would it be possible for you to provide images of the surface that you are testing the devices on? What depth configuration are you using when activating the pixel sensor?
GspLi
August 11, 2024, 7:18am
6
I was testing on multiple surfaces. This is the image of the surface that I used to generate the earlier PCD. Generally, I found
ML-captured PCDs are dense but have much noise compared to HL.
I checked it with the old MLSDK and the new OpenXR, and the results are the same.
I could not find any differences between Distorted and Undistorted PCD using the new OpenXR API. But there were differences in the old version.
FOV is not adjustable which also causes a lot of noise issues.
Here is the snippet of the code:
public uint targetStream
{
get { return DepthRange > 1.0f ? (uint)0 : (uint)1; }
}
public enum LongRangeUpdateRate
{
OneFps = 1, FiveFps = 5
}
public enum ShortRangeUpdateRate
{
FiveFps = 5, ThirtyFps = 30, SixtyFps = 60
}
[Header("ShortRange =< 1m")] private ShortRangeUpdateRate SRUpdateRate = ShortRangeUpdateRate.FiveFps;
[Header("LongRange > 1m")] private LongRangeUpdateRate LRUpdateRate = LongRangeUpdateRate.FiveFps;
private MagicLeapPixelSensorFeature pixelSensorFeature;
private void InitializePixelSensorFeature()
{
// Get the Magic Leap Pixel Sensor Feature from the OpenXR settings
pixelSensorFeature = OpenXRSettings.Instance.GetFeature<MagicLeapPixelSensorFeature>();
if (pixelSensorFeature == null || !pixelSensorFeature.enabled)
{
progressHandler.Fail("Pixel Sensor Feature Not Found or Not Enabled!");
return;
}
// Permissions.RequestPermission(UnityEngine.Android.Permission.Camera);
Permissions.RequestPermission(Permissions.DepthCamera,
(permission) =>
{
if (permission.Contains(Permissions.DepthCamera))
depthSensorId = FindAndInitializeSensor("/pixelsensor/depth/center");
},
(permission) =>
{
progressHandler.Fail("Permission Denied: " + permission);
});
progressHandler.Log("InitializePixelSensorFeature finished");
}
private PixelSensorId FindAndInitializeSensor(string pixelSensorPath)
{
var sensors = pixelSensorFeature.GetSupportedSensors();
PixelSensorId sensorId = sensors.Find(x => x.XrPathString == pixelSensorPath);
if (pixelSensorFeature.CreatePixelSensor(sensorId))
{
progressHandler.Log("Sensor created successfully " + pixelSensorPath);
}
else
{
progressHandler.Fail("Failed to create sensor " + pixelSensorPath);
throw new Exception("Failed to create sensor " + pixelSensorPath);
}
StartCoroutine(StartSensorStream(sensorId));
return sensorId;
}
private PixelSensorCapabilityType[] _targetCapabilityTypes = new[]
{
PixelSensorCapabilityType.UpdateRate,
PixelSensorCapabilityType.Format,
PixelSensorCapabilityType.Resolution,
PixelSensorCapabilityType.Depth,
};
// Coroutine to configure stream and start sensor streams
// targetStream: 0 for Long, 1 for short
private IEnumerator StartSensorStream(PixelSensorId sensorId)
{
pixelSensorFeature.GetPixelSensorCapabilities(sensorId, targetStream, out var capabilities);
foreach (var pixelSensorCapability in capabilities)
{
if (!_targetCapabilityTypes.Contains(pixelSensorCapability.CapabilityType))
{
continue;
}
// More details about the capability
if (pixelSensorFeature.QueryPixelSensorCapability(sensorId, pixelSensorCapability.CapabilityType, targetStream, out PixelSensorCapabilityRange range) && range.IsValid)
{
if (range.CapabilityType == PixelSensorCapabilityType.UpdateRate)
{
var configData = new PixelSensorConfigData(range.CapabilityType, targetStream);
configData.IntValue = DepthRange > 1 ? (uint)LRUpdateRate : (uint)SRUpdateRate;
pixelSensorFeature.ApplySensorConfig(sensorId, configData);
}
else if (range.CapabilityType == PixelSensorCapabilityType.Format)
{
var configData = new PixelSensorConfigData(range.CapabilityType, targetStream);
configData.IntValue = (uint)range.FrameFormats[UseRawDepth ? 1 : 0];
pixelSensorFeature.ApplySensorConfig(sensorId, configData);
}
else if (range.CapabilityType == PixelSensorCapabilityType.Resolution)
{
var configData = new PixelSensorConfigData(range.CapabilityType, targetStream);
configData.VectorValue = range.ExtentValues[0];
pixelSensorFeature.ApplySensorConfig(sensorId, configData);
}
else if (range.CapabilityType == PixelSensorCapabilityType.Depth)
{
var configData = new PixelSensorConfigData(range.CapabilityType, targetStream);
configData.FloatValue = DepthRange;
pixelSensorFeature.ApplySensorConfig(sensorId, configData);
}
}
}
var configureOperation = pixelSensorFeature.ConfigureSensor(sensorId, new[] { targetStream });
yield return configureOperation;
if (!configureOperation.DidOperationSucceed)
{
progressHandler.Fail("Failed to configure sensor.");
}
progressHandler.Log("Sensor configuration finished.");
// Obtain the supported metadata types from the sensor
Dictionary<uint, PixelSensorMetaDataType[]> supportedMetadataTypes = new();
if (pixelSensorFeature.EnumeratePixelSensorMetaDataTypes(sensorId, targetStream, out var metaDataTypes))
{
supportedMetadataTypes[targetStream] = metaDataTypes;
}
// Start the sensor with the default configuration and specify that all of the meta data should be requested.
PixelSensorAsyncOperationResult sensorStartAsyncResult =
pixelSensorFeature.StartSensor(sensorId, new[] { targetStream }, supportedMetadataTypes);
yield return sensorStartAsyncResult;
if (!sensorStartAsyncResult.DidOperationSucceed)
{
progressHandler.Fail("Failed to start sensor.");
}
progressHandler.Log("Sensor start finished. Monitoring data...");
}
1 Like
Thank you for that information, I have shared your feedback with our voice of customer team.
Note our depth sensor API provides the data without additional filtering. Glossy or dark surfaces will also affect the data since those materials interact with the light emitted by the sensor.
GspLi
August 12, 2024, 6:05am
8
It would be really helpful if the Magic Leap team could provide configuration guidelines and code for Depth to PCD.
GspLi
August 20, 2024, 5:06pm
9
Hi @kbabilinski , any update on this?
I have requested the feedback on your behalf which has now been marked as a feature request. If you wish to use the filtered depth data, you can use the Meshing feature instead that tries to reduce the noise.
I do not have any updates regarding your feature request at this time. You may want to use the Meshing feature instead of the raw depth for development.
