ML Spectator is not working with OpenXR

Unity Editor version: 2022.3.29f1
ML2 OS version: 1.7.0
Unity SDK version: 2.2.0
Host OS: Windows
OpenXR: 1.11.0
ML Spectator: 1.3.3

Hello,

I'm seeking help with an issue I've encountered while implementing ML Spectator in an OpenXR based Unity project.

The issue is as follows: After I use the ML Spectator App and scan the provided Aruco marker with an iOS device, Magic Leap fails to recognize it and gets stuck in the "Look at the marker" phase. After debugging, I've found that the **MLMarkerTracker.OnMLMarkerTrackerResultsFoundArray** callback always returns an empty array (**MarkerData[] dataArray** ), hence the marker is never detected.

image2

It's important to note that the ML Spectator marker detection utilizes the ML SDK's marker detection, not the OpenXR one.

OpenXR: Marker Understanding API Overview | MagicLeap Developer Documentation

ML SDK: Marker Tracker API Overview | MagicLeap Developer Documentation

The OpenXR marker detection API works fine. I've replaced the ML Spectator marker detector with the OpenXR Aruco marker tracker, and it works, but it places the objects in a completely different position and orientation on the iOS device than on the Magic Leap. (XR Rig and CameraOffset is in the origo with no rotation)

Is there a way to get the original ML SDK Aruco tracker to work under OpenXR?

Or how can I configure the OpenXR Aruco tracker to place objects in the correct transform?

Thanks in advance for your assistance!

image

Hi @csabai.tamas.93,

Thank you for reaching out regarding this issue. I'm going to test our MLSDK setup with spectator. Are there other OpenXR features that you are looking to use alongside the Marker Tracker? Do you plan to use size estimation at all?

Best,

El

I was able to get the MLSDK Marker tracking working while working in OpenXR. You can turn on Magic leap in the Plug-in Providers settings page.

This will allow you to use the MLSDK alongside OpenXR. I will go ahead and submit feedback to our team to let them know about the issues with this and I will report back as soon as I learn more.

Try running the marker tracking example script to see if this works with your spectator as a workaround.

Best,

El

What are the lighting conditions in your environment like also?

Thanks for the reply.

The environment is a well lit office room.

Originally, the "Magic Leap" checkbox didn't show up for me in XR Plugin Management. But after adding the com.unity.xr.magicleap (7.0.0) package to the project, it appeared.

image

image

So now the original MLarucoMarkerDetector script (ML SDK) works for me, but all OpenXR features are lost (e.g., Hand Tracking). I can only get these working again if I remove the com.unity.xr.magicleap package and thus the Magic Leap checkbox disappears as well.

Furthermore, with this marker tracking, it also places the virtual objects on the iPhone in a completely different place than where I see them on the Magic Leap. (About 1 meter away and rotated completely differently, facing the ground)

So the primary issue is not with the marker tracking itself, as it can be made to function with both the ML SDK and OpenXR. Rather, the problem lies in the virtual objects being placed in entirely different locations in space than where they should be.

Magic Leap:

iOS Spectator:

Where is your main camera positioned in the editor? If it is not at (0,0,0) in world coordinates, it may be the cause of this offset issue.

Best,

El