I am currently attempting to make my OpenXR simulation with MRTK3 interactivity compatible with Magic Leap's Spectator App on iOS.
Currently, when attempting to use the spectator app with my sim, the sim builds and loads fine. It performs and functions as expected when not attempting to synch with the spectator app. When I do attempt to use the spectator app itself, I can find my device on the network, connect to it, and scan the ARUco/marker with the app on the phone. However, when attempting the next step in the pairing process, to scan the ARUco/marker with my headset, the headset is unable to process the ARUco. Breakpoint debugging shows that the sim isn't even attempting to process the marker, though scanning is enabled.
I want to make it clear that the MarkerUnderstanding tracking works completely fine with this setup. My QR codes that are used for my code's logic are recognized and properly used. It is just the marker tracking that comes from the Spectator App that does not work.
From previous threads, I know that the Magic Leap Spectator app is reliant on Magic Leap's XR plugin (com.unity.xr.magicleap), however, whenever I install and enable this plugin, my app crashes on startup, although I receive no build errors, and can enter play mode just fine in my Unity editor.
Is Spectator App incompatible with OpenXR and MRTK3? If not what could be wrong with my set up? If it is, are there any plans soon to fix compatibility issues?
Hi I spoke to some of the engineers and they said they were aware of an issue with mrtk, openxr, and spectator, but that it should have been cleared up by the latest versions. Could you try updating versions and let me know if that fixes it?
Ok, I got some further guidance. Could you try enabling Perception snapshots and see if that gets it working? if it does then there are probably some URP settings to check as well
Actually I was incorrect. Disable and remove the old plugin (Magic leap xr for unity). Additionally it was brought to my attention that there is a bug we are working with Unity to fix causing crashes with the latest "Magic Leap XR Plugin for Unity" version 7.1.0, that may be the root cause of the crashing problem here. You can check via the changelog
That did it! I only had the XR plugin installed for spectator, the app worked fine without it but the spectator functionality didn't work. Looks like enabling Perception Snapshots allowed the spectator to function.
However, this seems to conflict with my MarkerUnderstanding Functionality. The two seem to be mutually exclusive where if I scan for the spectator app, I can no longer scan my QR Code for the rest of my app's functionality. If I scan the QR Code, I cannot scan for spectator app, and also my QR Code position seems to be significantly warped rotation-wise, and I am unable to execute any of my rescan/recalibration functions. Any ideas for these issues? Glad we're making progress here.
There seem to be some issues with a few minor versions of unity. Others on the team reported issues with versions past 2022.3.11f1 with version 2022.3.33f1 fixing the issue. Could you try upgrading to version 2022.3.33f1 and see if that fixes the problem?
Looks like we're getting closer. Spectator seems to work fine in tandem with vertical QR Codes. However, my horizontal QR codes don't seem to function at all.
That is really interesting. I will ask if anyone has seen this. Just for clarity, are you saying that rotating the qr code 90 degrees makes it stop working?
Bumping because it has been a while. I'm wondering if something with my QRCode Marker Tracking is conflicting with other Spectator App's Marker Tracking. This shouldn't be the case because they both instantiate their own MarkerUnderstandingFeature and Detector objects, right? Or am I misunderstanding how OpenXR features work?
Check out this example project in MRTK. You should be able to use the marker tracker using the following example script in this project. If you are using the marker tracker, you can have only 1 marker tracker active at once.
using System.Collections.Generic;
using Unity.XR.CoreUtils;
using UnityEngine;
using UnityEngine.XR.OpenXR;
using MagicLeap.OpenXR.Features.MarkerUnderstanding;
public class MarkerTrackerExample : MonoBehaviour
{
[Tooltip("Set the XR Origin so that the marker appears relative to headset's origin. If null, the script will try to find the component automatically.")]
public XROrigin XROrigin;
[Tooltip("If Not Null, this is the object that will be created at the position of each detected marker.")]
public GameObject MarkerPrefab;
public ArucoType ArucoType = ArucoType.Dictionary_5x5_50;
public MarkerDetectorProfile DetectorProfile = MarkerDetectorProfile.Default;
private MarkerDetectorSettings _detectorSettings;
private MagicLeapMarkerUnderstandingFeature _markerFeature;
private readonly Dictionary<string, GameObject> _markerObjectById = new Dictionary<string, GameObject>();
private void OnValidate()
{
// Automatically find the XROrigin component if it's present in the scene
if (XROrigin == null)
{
XROrigin = FindAnyObjectByType<XROrigin>();
}
}
private void Start()
{
_markerFeature = OpenXRSettings.Instance.GetFeature<MagicLeapMarkerUnderstandingFeature>();
if (_markerFeature == null || _markerFeature.enabled == false)
{
Debug.LogError("The Magic Leap 2 Marker Understanding OpenXR Feature is missing or disabled enabled. Disabling Script.");
this.enabled = false;
return;
}
if (XROrigin == null)
{
Debug.LogError("No XR Origin Found, markers sample will not work. Disabling Script.");
this.enabled = false;
}
// Create the Marker Detector Settings
_detectorSettings = new MarkerDetectorSettings();
// Configure a generic detector with QR and Aruco Detector settings
_detectorSettings.QRSettings.EstimateQRLength = false;
_detectorSettings.QRSettings.QRLength = //0.5f; <---- Enter QR length here
_detectorSettings.ArucoSettings.EstimateArucoLength = true;
_detectorSettings.ArucoSettings.ArucoType = ArucoType;
_detectorSettings.MarkerDetectorProfile = DetectorProfile;
// We use the same settings on all 3 of the
// different detectors and target the specific marker by setting the Marker Type before creating the detector
// Create QRCode Detector
_detectorSettings.MarkerType = MarkerType.QR;
_markerFeature.CreateMarkerDetector(_detectorSettings);
}
private void OnDestroy()
{
if (_markerFeature != null)
{
_markerFeature.DestroyAllMarkerDetectors();
}
}
void Update()
{
// Update the marker detector
_markerFeature.UpdateMarkerDetectors();
// Iterate through all of the marker detectors
for (int i = 0; i < _markerFeature.MarkerDetectors.Count; i++)
{
// Verify that the marker detector is running
if (_markerFeature.MarkerDetectors[i].Status == MarkerDetectorStatus.Ready)
{
// Cycle through the detector's data and log it to the debug log
MarkerDetector currentDetector = _markerFeature.MarkerDetectors[i];
OnUpdateDetector(currentDetector);
}
}
}
private void OnUpdateDetector(MarkerDetector detector)
{
for (int i = 0; i < detector.Data.Count; i++)
{
string id = "";
float markerSize = .01f;
var data = detector.Data[i];
switch (detector.Settings.MarkerType)
{
case MarkerType.Aruco:
id = data.MarkerNumber.ToString();
markerSize = data.MarkerLength;
break;
case MarkerType.QR:
id = data.MarkerString;
markerSize = data.MarkerLength;
break;
case MarkerType.UPCA:
Debug.Log("No pose is given for marker type UPCA, Code value is " + data.MarkerString);
break;
}
if (!data.MarkerPose.HasValue)
{
Debug.Log("Marker Pose not estimated yet.");
return;
}
if (!string.IsNullOrEmpty(id) && markerSize > 0)
{
Debug.Log("New Marker found");
// If the marker ID has not been tracked create a new marker object
if (!_markerObjectById.ContainsKey(id))
{
// Create a primitive cube
if (MarkerPrefab)
{
GameObject newMarker = Instantiate(MarkerPrefab);
_markerObjectById.Add(id, newMarker);
}
else
{
GameObject newDefaultMarker = GameObject.CreatePrimitive(PrimitiveType.Cube);
_markerObjectById.Add(id, newDefaultMarker);
}
}
GameObject marker = _markerObjectById[id];
SetTransformToMarkerPose(marker.transform, data.MarkerPose.Value, markerSize);
}
}
}
private void SetTransformToMarkerPose(Transform marker, Pose markerPose, float markerSize)
{
Transform originTransform = XROrigin.CameraFloorOffsetObject.transform;
// Set the position of the marker. Since the pose is given relative to the XR Origin,
// we need to transform it to world coordinates.
marker.position = originTransform.TransformPoint(markerPose.position);
marker.rotation = originTransform.rotation * markerPose.rotation;
marker.localScale = new Vector3(markerSize, markerSize, markerSize);
}
}
