I’m developing an AR application on Magic Leap 2 using Unity (2022.3.22f1) with Magic Leap SDK 1.12.0. In my project, I display dynamic UI elements (like face bounding boxes) on a World Space Canvas placed in front of the user. Everything works functionally, but I’m running into a persistent stereo misalignment issue:
The UI elements (e.g., image boxes with labels) appear in different positions for the left and right eyes, creating a “ghosting” or double-vision effect. For each detected face, one eye sees the box in the correct place, while the other sees it slightly offset.
Notably, when closing one eye:
With the left eye open, the bounding box appears slightly to the left of the face.
With the right eye open, the bounding box appears slightly to the right of the face.
This suggests a stereo parallax issue — the UI elements are being rendered in slightly different positions for each eye, failing to converge at the correct world space location.
Some additional context:
The Canvas is set to World Space mode, with a fixed size (640x480) and positioned 1.5 meters in front of the Main Camera.
The World Space Canvas itself is repositioned in LateUpdate() to remain a fixed distance in front of the user’s head.
The bounding boxes are generated and updated once per image frame, based on the resolution of the RGB image used for detection.
I’m using a Main Camera from ML Rig, which is configured for stereo rendering by default.
The Canvas and all UI elements are placed in the same layer (UI), and the Canvas is assigned the correct Event Camera.
The boxes are basic Image + TextMeshPro prefabs instantiated under the canvas.
I’ve also tried enabling Additional Shader Channels: TexCoord1 and TexCoord2, but the issue remains.
What I’m looking for:
Is there a recommended way to ensure identical positioning of UI elements across both eyes in stereo mode?
Would rendering the UI for one eye only (monoscopic) be a viable workaround in this case?
There are a few things you can try to resolve this.
Updating the Canvas position every frame in LateUpdate() (or Update()) causes slight discrepancies between when Unity renders each eye and when you move the canvas.
Either make the canvas position fixed (don’t move it) or If you want to follow the camera, do it in OnPreRender() or use Unity’s Camera.onPreRender callback to ensure the position is correct just before rendering.
You may not be accounting or projecting the bounding boxes into world space properly.
You’re converting 2D face detection boxes (from a 640x480 RGB image) into UI element positions, but maybe not accounting for the projection from image space into stereo 3D world space.
Transform your 2D detections into proper 3D world positions using the camera intrinsics and Magic Leap’s calibration parameters.
Magic Leap provides access to the RGB camera’s intrinsics, including focal length and principal point. Use this to calculate the 3D ray through each bounding box center and place UI elements at a consistent depth
The way to obtain the inrinsics differs depending on what camera API you are using.
If you are using the PixelSensor API, here is the documentation to get you started on obtaining this data and using it for your computations.
Example code of the computation you may want to do:
Vector3 GetWorldPositionFromImagePoint(Vector2 imagePoint, float depthMeters)
{
// Get intrinsics
var intrinsics; //This will depend on what API you are using to access the cameras. You must set this accordingly
float fx = intrinsics.FocalLength.x;
float fy = intrinsics.FocalLength.y;
float cx = intrinsics.PrincipalPoint.x;
float cy = intrinsics.PrincipalPoint.y;
// Compute normalized device coordinates
float x = (imagePoint.x - cx) / fx;
float y = (imagePoint.y - cy) / fy;
Vector3 ray = new Vector3(x, -y, 1f).normalized; // Left-handed to right-handed, if needed
// Place at fixed depth
Vector3 localPosition = ray * depthMeters;
return Camera.main.transform.TransformPoint(localPosition);
}