Hologram Drift Issue When Tracking Object with Retroreflective Markers using Depth Camera (Raw data)

[kbabilinski]

Sorry to hear that you are running into this issue. The team is out of office today but you could look into making a minor change in the SDK itself and see if it resolves your issue. Possibly by modifying a function in the SDK to allow for an additional property to be passed into the Get Pose function. (Note we have not tested this workaround yet)

Move the Magic Leap Unity SDK into your project’s packages directory.

Make sure the Magic Leap Unity Package is imported into your project as a folder in your /packages/ directory. (You can skip this step if you are using the Magic Leap Examples Project)

1. Right-click the Magic Leap SDK folder in the Unity Editor and then select “Show in file explorer” (or equivalent)
   

   

2. Copy the com.magicleap.unitysdk directory and paste it into the <YourProject>/packages/ directory.
   

   

3. The contents of the folder can now be edited without reverting to the version that was embedded in the package.

Edit the Scripts

1. Open the MagicLeapPixelSensor.cs script.
   Located under: /Packages/com.magicleap.unitysdk/Runtime/OpenXR/PixelSensors/MagicLeapPixelSensor.cs

2. Duplicate the GetSensorPose function (lines 74-98) and modify it to accept a captureTime and uses this value instead of the next predicted display time.

public Pose GetSensorPose(Pose offset, long captureTime)
        {
            unsafe
            {
                var convertedOffsetPose = XrPose.GetFromPose(offset);
                if (sensorSpace == 0)
                {
                    var createSpaceInfo = new XrPixelSensorCreateSpaceInfo
                    {
                        Type = XrPixelSensorStructTypes.XrTypePixelSensorCreateSpaceInfoML,
                        Sensor = Handle,
                        Offset = convertedOffsetPose
                    };
                    var xrResult = NativeFunctions.XrCreatePixelSensorSpace(PixelSensorFeature.AppSession, ref createSpaceInfo, out sensorSpace);
                    if (!Utils.DidXrCallSucceed(xrResult, nameof(PixelSensorNativeFunctions.XrCreatePixelSensorSpace)))
                    {
                        return default;
                    }
                }

                var spaceInfoFunctions = PixelSensorFeature.SpaceInfoNativeFunctions;
                // The line below was updated to accept a capture time value.
                var pose = spaceInfoFunctions.GetUnityPose(sensorSpace, PixelSensorFeature.AppSpace, captureTime);
                return pose;
            }
        }

3. Open the MagicLeapPixelSensorFeatureAPI.cs script.
   Located under : /Packages/com.magicleap.unitysdk/Runtime/OpenXR/PixelSensors/MagicLeapPixelSensorFeatureAPI.cs

4. Duplicate the GetSensorPose function and modify the new version to accept a captureTime.

      public Pose GetSensorPose(PixelSensorId sensorType, long captureTime, Pose offset = default)
        {
            if (!IsSensorConnected(sensorType, out var sensor))
            {
                return default;
            }
            // The line below was modified to pass in the capture time value.
            return sensor.GetSensorPose(offset,captureTime);
        }

Edit your existing code

1. Finally, with these modifications made you can try to obtain the pose as you did previously but passing the capture time into the GetSensorPose function

if (pixelSensorFeature.GetSensorData(sensorId.Value, stream, out var frame, out var metaData,
        Allocator.Temp, shouldFlipTexture: true))
{
    // The line below was modified to pass in the Capture Time value
    Pose sensorPose = pixelSensorFeature.GetSensorPose(sensorId.Value, frame.CaptureTime, offset);
    Debug.Log("Sensor Pose:" + sensorPose);

    // Use Frame and Pose Data
    // Example : streamVisualizer.ProcessFrame(frame, metaData, sensorPose);
    
}