MLWorldCameraFrame.camera_pose to Unity mapping

Native Development
1

Hola,

ML2 OS version: 1.10.0
MLSDK version: 1.10.0
Host OS: Win

We use the World Camera images to reconstruct poses of our custom marker in the world camera frame. The mapping from the camera frame to the world frame using the MLWorldCameraFrame.camera_pose seems to work fine: When i walk around the marker its pose remains almost constant.

From within Unity we access the custom marker poses via an android AAR plugin. This all works but how do i convert from the "world co-ordinate system" referenced in the MLWorldCameraFrame docs to the respective Unity coordinate system ? I there any Magic Leap Native SDK (C API) function we can use for that ?

Btw. i have Zero (-0) Unity experience :wink:

Cheers

1 Like
2

If you are using Magic Leap SDK instead of the OpenXR plugin - you can try the following in Unity:

  1. Select the Main camera under the Magic Leap Rig in Unity.
  2. Make sure the XR Rig component does not have any offsets applied in the hierarchy or inspector.
  3. Look for the Magic Leap Camera Component in the Inspector.
  4. Disable the Recenter XR Origin on Start option.
  5. Then test the App again and see if the poses align properly.
3

Hi,
we are currently on a MRTK OpenXR project. The XR Rig is MRTK XR Rig - MagicLeap - OpenXR, which has no offset. There's a 1.6 camera offset on Y (the default one).
If I navigate to Magic Leap Camera Script (which seems to be deprecated) on the Main Camera, the Recenter XR Origin at Start is already set false.

What I did was to Camera.main.transform.TransformPoint() the real-world pose and then add a manual offset to fix the misalignment camera->eye lens center...

Is there a better (and more precise) way to do it?

Thanks

4

What type of offset are you seeing when you transform the point?

If you are using the MLSDK in conjunction with OpenXR you will need to make sure your application has the origin set to Unbounded Space to have the same origin as the Legacy MLSDK:

5

I'm measuring around:
x: 5mm to the right
y: 27.5mm up
z: 38mm forward

6

Do you feel like it roughly matches the pose of the sensor relative to the center of the two displays? Have you tried the unbounded space?

7

Not really,
And yes, I've tried Unbounded Spaces but nothing seems to change

8

A couple of things you could try:

Option 1: Pixel Sensor API (Recommended)
Have you looked into using the OpenXR Pixel Sensor API? It's the preferred way to access sensor data these days and plays nicely with OpenXR.

Option 2: Legacy MLSDK
If you're sticking with the older way, you may want to try using the legacy MLSDK workflow.

Testing if Unbounded Space is being set properly in Unity

To test if the unbounded space is being set properly in Unity, follow these steps:

  1. Create a new scene.
  2. Go to the Package Manager and import the XR Rig package from the Magic Leap SDK.
  3. In the inspector, set the XR Origin to "device".
  4. Place a 3D cube into your Unity scene, so that it can be viewed by the main camera
  5. Run the application and see if the virtual content shifts when you switch the tracking to unbounded.

Note: Both Perception Snapshots setting and the Reference Space feature need to be enabled in your Project's OpenXR Settings.

Some background information based on the MLCamera:

The MLCVCameraGetFramePose gives you the position of the camera in world origin coordinate system which is not a concept in OpenXR. However it happens to align with our implementation of msft_unbounded_ref_space so we take advantage of that to allow developers to express the position of the pose in OpenXR as they choose.

9

I have similar issue.
I'm using the legacy MLSDK default configurations:

  • Under the XR Rig game object the session origin tracking mode origin set to Device
  • In the Main Camera game object, Recenter XR Origin At Start is disabled
  • I configure the inputSubsystem tracking origin mode to unbound and called to TryRecenter as defined in the Unity (OpenXR) MLCamera documentations (link)

But when I call to MLCVCamera.GetFramePose(extras.VCamTimestamp, out var outTransform) I get the transformation between the frame and the origin where the magic leap powered up and not where my application started.

I tried to set the subsystem tracking mode to Device, but the outTransform was the same.

I don't know what else can be done

10

Do you mind creating a new post with the issue that you are experiencing and include the MLSDK and OS version you are using along with additional information about which XR Plugin-in provider you are using (OpenXR vs legacy XR Magic Leap )