Scanned marker's Pose at world origin

Give us as much detail as possible regarding the issue you’re experiencing:

Unity Editor version: Unity 6000.0.29f1
ML2 OS version: 1.12.0
Unity SDK version: 2.6.0
Host OS: (Windows)

Error messages from logs (syntax-highlighting is supported via Markdown): (No error messages since all have been resolved, although creating a detector in awake ensured the feature was never ready if done at application Boot, which would be nice as a compile-time error)

I make this post as suggested by support.

The Issue:
The issue is that a bunch of initial scan poses will always return world origin (0,0,0) .
There is a way to bypass this but doesn’t really solve the issue.
I have seen other posts about this but no .solutions

Solution (in airquotes cause it just bypasses the issue)

  • Use the documentation’s sample code as boilerplate
  • Instruct the user to move around (depth wise, but also on the world’s horizontal plane) while looking at the QR
  • Add a list of scanned poses with a defined accuracy (in my case 100 stable scans are required)
  • Average out the 100 scans, after removing any outliers and after removing any scans with 0,0,0 position values

This gives us a pretty decent pose of the QR.

I can also recreate the issue of the marker not being scanned (or perhaps the pose being 0,0,0) until I move my head around a bunch while looking at the QR in the MarkerUnderstandingSample.

I am asking if the developers are aware of this issue or if other people have found a solution for this rather than a bypass.

I’ll drop my current workaround below here.
I can make it a bit more decoupled if requested so it becomes more easily copy-pastable, if not requested I’ll leave it as is since I’m on a limited budget for this project.
So for now just remove the QrProgressManager dependency.

I’ve added a placeholder Monobehaviour that functions as a driver for the scanner class:
Unless someone has a nice solution so we can let the scanning logic flow be entirely independent of a MonoBehaviour here, would be quite nice.

public class MarkerScannerDriver : MonoBehaviour
{
    private IMarkerScanner _scanner;

    private void Awake()
    {
        _scanner = new Ml2MarkerScanner();
        _scanner.OnMarkerDetected += HandleDetected;
    }

    private void Update()
    {
        if (_scanner.IsScanning)
            _scanner.PollMarkers();
    }
    
    [ContextMenu("Start Scanning")]
    public void StartScanning()
    {
        _scanner.StartScanning();
    }

    [ContextMenu("Stop Scanning")]
    public void StopScanning()
    {
        _scanner.StopScanning();
    }
    
    private void HandleDetected(string arg1, Pose arg2, float arg3)
    {
        Debug.Log($"Detected QR: Payload: {arg1} Pose: {arg2} Size: {arg3}");
    }
}
using System;
using UnityEngine;

namespace Simultria.QRScanning
{
    public interface IMarkerScanner
    {
        public event Action<string, Pose, float> OnMarkerDetected;
        public bool IsScanning { get; }
        public void StartScanning();
        public void StopScanning();
        public void PollMarkers();
    }
}

for some reason I see no code highlighting in the below preview here, sorry

using System;
using System.Collections.Generic;
using System.Linq;
using MagicLeap.OpenXR.Features.MarkerUnderstanding;
using Simultria.Feedbacks;
using UnityEngine;
using UnityEngine.XR.OpenXR;

namespace Simultria.QRScanning
{
    public class Ml2MarkerScanner : IMarkerScanner
    {
        public event Action<string, Pose, float> OnMarkerDetected;

        private readonly MagicLeapMarkerUnderstandingFeature _feature;
        private readonly MarkerDetectorSettings _settings;
        private readonly MarkerDetector _qrDetector;

        private readonly Dictionary<string, Pose> _lastPoses = new Dictionary<string, Pose>();
        private readonly Dictionary<string, int> _stableCounts = new Dictionary<string, int>();
        private readonly Dictionary<string, List<Pose>> _recentStablePoses = new Dictionary<string, List<Pose>>();

        private const int STABLE_THRESHOLD = 10;
        private const float POSITION_TOLERANCE = 0.01f;
        private const float ROTATION_TOLERANCE = 1.0f;
        private const int AVERAGE_OVER = 8;
        private const int BUFFER_SIZE = 120;

        public bool IsScanning { get; private set; }

        public Ml2MarkerScanner()
        {
            _feature = OpenXRSettings.Instance.GetFeature<MagicLeapMarkerUnderstandingFeature>();
            if (_feature == null || !_feature.enabled)
            {
                Debug.LogError("MagicLeapMarkerUnderstandingFeature is not enabled in OpenXR settings!");
                return;
            }

            _settings = new MarkerDetectorSettings
            {
                    MarkerDetectorProfile = MarkerDetectorProfile.Default, MarkerType = MarkerType.QR
            };
            _settings.QRSettings.QRLength = 0.115f;
            _settings.QRSettings.EstimateQRLength = true;
            _qrDetector = _feature.CreateMarkerDetector(_settings);
        }

        public void StartScanning()
        {
            IsScanning = true;
        }

        public void StopScanning()
        {
            IsScanning = false;
        }

        public void PollMarkers()
        {
            if (_qrDetector == null || !IsScanning)
                return;

            _feature.UpdateMarkerDetectors();

            for (var i = 0; i < _qrDetector.Data.Count; i++)
            {
                MarkerData data = _qrDetector.Data[i];
                ProcessMarkerData(data);
            }
        }

        private void ProcessMarkerData(MarkerData data)
        {
            if (!data.MarkerPose.HasValue)
                return;

            Pose newPose = new Pose(data.MarkerPose.Value.position, data.MarkerPose.Value.rotation);
            if (newPose.position == Vector3.zero)
                return;

            var markerString = data.MarkerString;
            var size = data.MarkerLength > 0 ? data.MarkerLength : _settings.QRSettings.QRLength;

            var wasStable = IsMarkerStable(markerString);
            UpdateStability(markerString, newPose);

            if (!IsMarkerStable(markerString))
                return;

            var poses = GetOrCreatePoseBuffer(markerString);

            if (poses.Count < BUFFER_SIZE)
                AddStablePose(markerString, poses, newPose);

            if (poses.Count == BUFFER_SIZE)
                TryStabilizeMarker(markerString, poses, size, wasStable);
        }

        private bool IsMarkerStable(string markerString) =>
                _stableCounts.TryGetValue(markerString, out var stableFrames) && stableFrames >= STABLE_THRESHOLD;

        private void UpdateStability(string markerString, Pose newPose)
        {
            if (_lastPoses.TryGetValue(markerString, out Pose lastPose))
            {
                var posDist = Vector3.Distance(lastPose.position, newPose.position);
                var rotDist = Quaternion.Angle(lastPose.rotation, newPose.rotation);

                if (posDist < POSITION_TOLERANCE && rotDist < ROTATION_TOLERANCE)
                    _stableCounts[markerString]++;
                else
                    _stableCounts[markerString] = 0;
            }
            else
                _stableCounts[markerString] = 0;

            _lastPoses[markerString] = newPose;
        }

        private List<Pose> GetOrCreatePoseBuffer(string markerString)
        {
            if (!_recentStablePoses.TryGetValue(markerString, out var poses))
            {
                poses = new List<Pose>();
                _recentStablePoses[markerString] = poses;
            }
            return poses;
        }

        private void AddStablePose(string markerString, List<Pose> poses, Pose newPose)
        {
            poses.Add(newPose);
            var progress = poses.Count / (float)BUFFER_SIZE;
            QrProgressManager.Instance.UpdateProgress(markerString, progress, newPose);
        }

        private void TryStabilizeMarker(string markerString, List<Pose> poses, float size, bool wasStable)
        {
            var filteredPoses = RemoveOutlierPoses(poses);

            if (filteredPoses.Count >= AVERAGE_OVER)
            {
                var recentFiltered = filteredPoses
                                    .Skip(filteredPoses.Count - AVERAGE_OVER)
                                    .Take(AVERAGE_OVER)
                                    .ToList();

                Vector3 avgPos = AveragePosition(recentFiltered);
                Quaternion avgRot = AverageQuaternions(recentFiltered);

                Pose averagedPose = new Pose(avgPos, avgRot);
                OnMarkerDetected?.Invoke(markerString, averagedPose, size);

                QrProgressManager.Instance.HideProgress(markerString);
                poses.Clear();
            }
            else
            {
                poses.Clear();
                QrProgressManager.Instance.HideProgress(markerString);
            }
        }

        public void ResetAllMarkers()
        {
            _lastPoses.Clear();
            _stableCounts.Clear();
            _recentStablePoses.Clear();
            QrProgressManager.Instance.HideAllProgress();
        }

        private static Vector3 AveragePosition(List<Pose> poses)
        {
            Vector3 sum = poses.Aggregate(Vector3.zero, (current, p) => current + p.position);
            return sum / poses.Count;
        }

        private static Quaternion AverageQuaternions(List<Pose> poses)
        {
            if (poses.Count == 1)
                return poses[0].rotation;

            Quaternion avg = poses[0].rotation;
            for (var i = 1; i < poses.Count; i++)
            {
                var t = 1f / (i + 1);
                avg = Quaternion.Slerp(avg, poses[i].rotation, t);
            }
            return avg;
        }

        private static List<Pose> RemoveOutlierPoses(List<Pose> poses)
        {
            if (poses.Count < 3)
                return new List<Pose>(poses);

            Vector3 median = AveragePosition(poses);
            var ordered = poses.OrderBy(p => (p.position - median).sqrMagnitude).ToList();
            var remove = Mathf.Max(1, poses.Count / 10);
            return ordered.Skip(remove).Take(poses.Count - remove * 2).ToList();
        }
    }
}
1 Like

Hey @j.g.hoevenberg,

Welcome to the Magic Leap Developer Forums!

I think the main issue you’re describing is being caused by the following line:

_settings.QRSettings.EstimateQRLength = true;

This requires the user to move their head around until the Magic Leap 2 is able to triangulate the detected poses so it can estimate the length of the QR code.

You can try setting this flag to false, but it will require you to know the size of the marker before running the app.

I assume that you are reporting that even with if (!data.MarkerPose.HasValue) and if(data.MarkerLength > 0), invalid poses are being provided. In this case, I can go ahead and file this as a bug!

In general I think your implementation is correct. You can try setting the EstimateQRLength flag to false, or you can continue doing what you are already doing, sampling poses until you get a stable pose.

Best,
Corey