No Metadata returned from GetSensorData

[ccroncc8231]

I am trying to get camera intrinsics but GetSensorData() is not giving any metadata in my app.
DebugLog shows metaData.Length is 0

The device can correctly return and output metadata in another example app. I have no idea what might be the reason. any idea on what I may look into?

public void Capture(StringBuilder frameDataText, out string datetimeString)
            {
                
                PixelSensorPinholeIntrinsics pinholeIntrinsics;
                datetimeString = "";
                if (PixelSensorFeature.GetSensorStatus(SensorId) ==
                   PixelSensorStatus.Started)
                {
                    if (!PixelSensorFeature.GetSensorData(SensorId, ConfiguredStream, out PixelSensorFrame frame, out PixelSensorMetaData[] metaData, Allocator.Temp))
                    {
                        return;
                    }
                    foreach (var meta in metaData)
                    {
                        Debug.Log("meta:" + meta);
                    }
                     if (PixelSensorFeature.GetSensorData(SensorId, ConfiguredStream, out frame,out metaData, Allocator.Temp, shouldFlipTexture: false))
                    {
                      Debug.Log(SensorId + " | " + metaData.Length + " | " + metaData);
                     some capture function}
}}
...........................

[kbabilinski]

Please see the Samples and Examples in the Magic Leap example project and the Magic Leap Developer portal.

[ccroncc8231]

Hi Thank you very much for your reply!

I tried our Unity examples and read documentation already. They are not giving me any clue why no metaData is returned. I modified the PixelSensorExample.cs like this in the Unity examples to print the returned metadata, but still got nothing, the metadata.Length shown in debug log is 0

public void UpdateVisualizer(StringBuilder frameDataText)
        {
            if (!PixelSensorFeature.GetSensorData(SensorId, ConfiguredStream, out var frame, out PixelSensorMetaData[] metadata, Allocator.Temp))
            {
                return;
            }
            string debugString = "";
            foreach (var meta in metadata)
            {
                debugString += meta;
                debugString += "|";
            }
            Debug.Log(metadata.Length+"|"+debugString);
            //asdas
            frameDataText.AppendLine("");
            frameDataText.AppendLine($"Sensor: {SensorId.SensorName}");
            frameDataText.AppendLine($"Frame Type: {frame.FrameType}");
            frameDataText.AppendLine($"Frame Valid: {frame.IsValid}");
            frameDataText.AppendLine($"Capture Time: {frame.CaptureTime}");
            frameDataText.AppendLine($"Frame Plane Count: {frame.Planes.Length}");
            Visualizer.ProcessFrame(in frame);
        }

[kbabilinski]

Which example script are you using? Do you mind posting the link to the original script and your modified one ? I think we have a few examples on the developer portal that log the metadata after configuring the sensor but I might be wrong.

[ccroncc8231]

Hi, I am using the “Unity examples” v2.6.0 downloaded from Magic Leap Hub 3.

I only modified the script named “PixelSensorExample.cs” in scene “PixelSensor”. From line 341 to line 360. The original one is:

public void UpdateVisualizer(StringBuilder frameDataText)
        {
            if (!PixelSensorFeature.GetSensorData(SensorId, ConfiguredStream, out var frame, out _, Allocator.Temp))
            {
                return;
            }
            frameDataText.AppendLine("");
            frameDataText.AppendLine($"Sensor: {SensorId.SensorName}");
            frameDataText.AppendLine($"Frame Type: {frame.FrameType}");
            frameDataText.AppendLine($"Frame Valid: {frame.IsValid}");
            frameDataText.AppendLine($"Capture Time: {frame.CaptureTime}");
            frameDataText.AppendLine($"Frame Plane Count: {frame.Planes.Length}");
            Visualizer.ProcessFrame(in frame);
        }

I simple added a debug log to see if it is returning metadata

public void UpdateVisualizer(StringBuilder frameDataText)
        {
            if (!PixelSensorFeature.GetSensorData(SensorId, ConfiguredStream, out var frame, out PixelSensorMetaData[] metadata, Allocator.Temp))
            {
                return;
            }
            string debugString = "";
            foreach (var meta in metadata)
            {
                debugString += meta;
                debugString += "|";
            }
            Debug.Log(metadata.Length+"|"+debugString);
            //asdas
            frameDataText.AppendLine("");
            frameDataText.AppendLine($"Sensor: {SensorId.SensorName}");
            frameDataText.AppendLine($"Frame Type: {frame.FrameType}");
            frameDataText.AppendLine($"Frame Valid: {frame.IsValid}");
            frameDataText.AppendLine($"Capture Time: {frame.CaptureTime}");
            frameDataText.AppendLine($"Frame Plane Count: {frame.Planes.Length}");
            Visualizer.ProcessFrame(in frame);
        }

As for my project, I took reference from API Overview | MagicLeap Developer Documentation , the example usage part of query sensor data.

The following is part of my code in my project. Frame data is outputed correctly but not metadata. If you need extra information please ask. I have no idea on what I should look into. Thank you very much!

public List<Vector3> Capture(StringBuilder frameDataText, out string datetimeString)
            {
                List<Vector3> depthPoints = new();
                PixelSensorPinholeIntrinsics pinholeIntrinsics;
                datetimeString = "";
                if (PixelSensorFeature.GetSensorStatus(SensorId) ==
                   PixelSensorStatus.Started)
                {
                    if (!PixelSensorFeature.GetSensorData(SensorId, ConfiguredStream, out PixelSensorFrame frame, out PixelSensorMetaData[] metaData, Allocator.Temp))
                    {
                        return depthPoints;
                    }
                    foreach (var meta in metaData)
                    {
                        Debug.Log("meta:" + meta);
                    }
                    if (PixelSensorFeature.GetSensorData(SensorId, ConfiguredStream, out frame,
                            out metaData, Allocator.Temp, shouldFlipTexture: false))
                    {
                        Debug.Log(SensorId + " | " + metaData.Length + " | " + metaData);
                        frameDataText.AppendLine("[Metadata]");
                        frameDataText.AppendLine($"Capture Time: {frame.CaptureTime}");
                        frameDataText.AppendLine($"Sensor: {SensorId.SensorName}");
                        frameDataText.AppendLine($"Frame Type: {frame.FrameType}");
                        frameDataText.AppendLine($"Frame Valid: {frame.IsValid}");
                        frameDataText.AppendLine($"Frame Plane Count: {frame.Planes.Length}");
                        Debug.Log("GetStringFromMetaData");
                        GetStringFromMetaData(in metaData, in frameDataText, out pinholeIntrinsics);
                        Debug.Log("pinholeIntrinsics: " + pinholeIntrinsics);
                        // pose
                        frameDataText.AppendLine($"[Extrinsics]");
                        var frameRotation = PixelSensorFeature.GetSensorFrameRotation(SensorId);
                        Pose sensorPose;
                        if (xrOrigin != null)
                        {
                            Pose offset = new Pose(
                            xrOrigin.CameraFloorOffsetObject.transform.position,
                            xrOrigin.transform.rotation);
                            sensorPose = PixelSensorFeature.GetSensorPose(SensorId, frame.CaptureTime, offset);
                        }
                        else
                        {
                            sensorPose = PixelSensorFeature.GetSensorPose(SensorId);
                        }
                        frameDataText.AppendLine($"Sensor Pose Position: {sensorPose.position.x:F7},{sensorPose.position.y:F7},{sensorPose.position.z:F7}");
                        frameDataText.AppendLine($"Sensor Pose Rotation: {sensorPose.rotation.x:F8},{sensorPose.rotation.y:F8},{sensorPose.rotation.z:F8},{sensorPose.rotation.w:F8}");
                        frameDataText.AppendLine($"Frame Rotation: {frameRotation}");

                        var confidenceBuffer = metaData.OfType<PixelSensorDepthConfidenceBuffer>().FirstOrDefault();
                        var flagBuffer = metaData.OfType<PixelSensorDepthFlagBuffer>().FirstOrDefault();
                        uint height, width;
                        Debug.Log("ProcessFrame");
                        var texture = ProcessFrame(in frame, in frameDataText, in confidenceBuffer, in flagBuffer, out height, out width, out string datetime);
                        datetimeString = datetime;
                        Debug.Log("resolution: " + height + " | " + width);
                        if (frame.FrameType == PixelSensorFrameType.Depth32)
                        {
                            //something to generate PCD from depth image
                        }
                        if (frame.FrameType == PixelSensorFrameType.DepthRaw)
                        {
                            //something else
                        }
                    }
                }
                return depthPoints;
            }

[kbabilinski]

I think the issue might be that you are calling the following function twice.

 PixelSensorFeature.GetSensorData(...)

Here is an example from one of my scripts. It’s in an Ienumerator because I was controlling when to enable/disable the polling that way.

private System.Collections.IEnumerator DepthStreamLoop()
{
    while (IsSensorRunning &&
           pixelSensor.GetSensorStatus(sensorId.Value) == PixelSensorStatus.Started)
    {
        if (!pollSensorData) { yield return null; continue; }

        if (!pixelSensor.GetSensorData(
                sensorId.Value, TargetStream,
                out var frame,
                out var meta,
                Allocator.Temp,
                shouldFlipTexture: true))
        {
            yield return null; continue;
        }

        if (!frame.IsValid || frame.Planes.Length == 0)
        {
            yield return null; continue;
        }

      
        OnDepthDataReceived?.Invoke(frame, meta);
        yield return null;
    }
    pollingLoop = null;
}

[ccroncc8231]

Thank you very much for your reply and your example.
The problem existed before I tried to call GetSensorData() twice. I wanted to see if calling again would work so I added a second one. As you can see, a Debug.Log() is called immediately after the first GetSensorData(). It is logging nothing.

I looked into the definition of GetSensorData() in MagicLeapPixelSensor.cs(magic leap unity sdk)

Debug.Log(requestedMetaData.Count + "| requestedMetaData.Count");

And added this debug.log in it. As a result the number of requestMetaData is 0. How can buffer.RequestedMetaDataTypes return nothing while the frame data is all good?

public unsafe bool GetSensorData(uint streamIndex, out XrPixelSensorFrame* sensorFrame, out PixelSensorMetaData[] metaData, Allocator allocator, long timeout, bool shouldFlipTexture)
        {
            bool FrameErrorResultCheck(XrResult result)
            {
                return result is XrResult.Success or XrResult.TimeoutExpored;
            }

            sensorFrame = null;
            metaData = Array.Empty<PixelSensorMetaData>();
            var buffer = StreamBuffers[streamIndex];
            var getInfo = new XrPixelSensorDataGetInfo
            {
                Type = XrPixelSensorStructTypes.XrTypePixelSensorDataGetInfoML,
                Stream = streamIndex,
                LastCaptureTime = buffer.LastCaptureTime,
                Timeout = timeout
            };
            var data = new XrPixelSensorData
            {
                Type = XrPixelSensorStructTypes.XrTypePixelSensorDataML,
                Next = IntPtr.Zero
            };
            var requestedMetaData = buffer.RequestedMetaDataTypes;

            var chainStart = &data.Next;
            var metaDataContainer = XrMetadataContainer.Create();
            Debug.Log(requestedMetaData.Count + "| requestedMetaData.Count");
            foreach (var currentMetadata in requestedMetaData)
            {
                switch (currentMetadata)
                {
                    case PixelSensorMetaDataType.ExposureTime:
                        *chainStart = new IntPtr(&metaDataContainer.ExposureTime);
                        chainStart = &metaDataContainer.ExposureTime.Next;
                        break;
                    case PixelSensorMetaDataType.AnalogGain:
                        *chainStart = new IntPtr(&metaDataContainer.AnalogGain);
                        chainStart = &metaDataContainer.AnalogGain.Next;
                        break;
                    case PixelSensorMetaDataType.DigitalGain:
                        *chainStart = new IntPtr(&metaDataContainer.DigitalGain);
                        chainStart = &metaDataContainer.DigitalGain.Next;
                        break;
                    case PixelSensorMetaDataType.PinholeCameraModel:
                        *chainStart = new IntPtr(&metaDataContainer.PinholeIntrinsics);
                        chainStart = &metaDataContainer.PinholeIntrinsics.Next;
                        break;
                    case PixelSensorMetaDataType.FishEyeCameraModel:
                        *chainStart = new IntPtr(&metaDataContainer.FisheyeIntrinsics);
                        chainStart = &metaDataContainer.FisheyeIntrinsics.Next;
                        break;
                    case PixelSensorMetaDataType.DepthFrameIllumination:
                        *chainStart = new IntPtr(&metaDataContainer.DepthFrameIllumination);
                        chainStart = &metaDataContainer.DepthFrameIllumination.Next;
                        break;
                    case PixelSensorMetaDataType.DepthConfidenceBuffer:
                        *chainStart = new IntPtr(&metaDataContainer.DepthConfidenceBuffer);
                        chainStart = &metaDataContainer.DepthConfidenceBuffer.Next;
                        break;
                    case PixelSensorMetaDataType.DepthFlagBuffer:
                        *chainStart = new IntPtr(&metaDataContainer.DepthFlagBuffer);
                        chainStart = &metaDataContainer.DepthFlagBuffer.Next;
                        break;
                    default:
                        throw new ArgumentOutOfRangeException();
                }
            }

            var xrResult = NativeFunctions.XrGetPixelSensorData(Handle, ref getInfo, out buffer.SensorBuffer, out data);
            buffer.LastCaptureTime = data.CaptureTime;
            if (!Utils.DidXrCallSucceed(xrResult, nameof(PixelSensorNativeFunctions.XrGetPixelSensorData), FrameErrorResultCheck))
            {
                return false;
            }

            sensorFrame = (XrPixelSensorFrame*)data.Frame;
            metaData = new PixelSensorMetaData[requestedMetaData.Count];
            for (var i = 0; i < requestedMetaData.Count; i++)
            {
                metaData[i] = requestedMetaData[i] switch
                {
                    PixelSensorMetaDataType.ExposureTime => metaDataContainer.ExposureTime.GetMetaData(allocator),
                    PixelSensorMetaDataType.AnalogGain => metaDataContainer.AnalogGain.GetMetaData(allocator),
                    PixelSensorMetaDataType.DigitalGain => metaDataContainer.DigitalGain.GetMetaData(allocator),
                    PixelSensorMetaDataType.PinholeCameraModel => metaDataContainer.PinholeIntrinsics.GetMetaData(allocator),
                    PixelSensorMetaDataType.FishEyeCameraModel => metaDataContainer.FisheyeIntrinsics.GetMetaData(allocator),
                    PixelSensorMetaDataType.DepthFrameIllumination => metaDataContainer.DepthFrameIllumination.GetMetaData(allocator),
                    PixelSensorMetaDataType.DepthConfidenceBuffer => metaDataContainer.DepthConfidenceBuffer.GetMetaData(allocator, shouldFlipTexture, data.CaptureTime),
                    PixelSensorMetaDataType.DepthFlagBuffer => metaDataContainer.DepthFlagBuffer.GetMetaData(allocator, shouldFlipTexture, data.CaptureTime),
                    _ => throw new ArgumentOutOfRangeException()
                };
            }

            return true;
        }

[ccroncc8231]

Actually it was a small mistake.

I may have copied code from two different pages so that I didn’t pass all other metadata type apart from depthflag and depthconfidence when I call StartSensor().

After adding those types to the PixelSensorMetaDataType array I passed into StartSensor() the problem is well solved.