Unable to get physical camera pose

logan · November 9, 2023, 6:19am

Give us as much detail as possible regarding the issue you're experiencing.

Unity Editor version:
ML2 OS version:
MLSDK version:
Host OS: (Windows/MacOS)

Error messages from logs (syntax-highlighting is supported via Markdown):

I'm using camera feed for precise CV work where I need very correct locations of pixels for ray casting etc. So I'm following Intrinsic/Extrinsic Parameters | MagicLeap Developer Documentation to get the info to transform the results. The distortion correction seems to be doing its job (tough to be 100% sure) but when I try to get the camerapose, I hit catastrophic errors.

By way of example, when I uncomment in the following code, suddenly my app won't keep rendering past a few frames

private void SetCapturePose(MLCameraBase.ResultExtras resultextras)
		{
			Debug.Log($"Capture: Starting pose finder. \n" +
			          $"Timestamp: {resultextras.VCamTimestamp} \n" +
			          $"Intrinsics: {resultextras.Intrinsics?.ToString() ?? "null"}");
			
			//Set pose info for later use in correction and raycasting. 
			Transform camTform = Cam.transform;
			var cameraPose = new Pose(camTform.position, camTform.rotation);
			
			// if (MLCVCamera.GetFramePose(resultextras.VCamTimestamp, out Matrix4x4 cameraTransform).IsOk)
			// 	CapturePose = new Pose(cameraTransform.GetPosition(), cameraTransform.rotation);
			// else
			// 	Debug.LogError("Capture: Attempting to get pose from resultExtras failed: VCamTimestamp is 0");
			
			Debug.Log($"Camera pose: {cameraPose} \n" +
			          $"Capture pose: {CapturePose}");
		}

I'm wondering if the sample code is missing context for how we need to get at that information. It also seems like the informaiton I'm looking for (the pose of the physical camera) should have a static relationship to the virtual camera so I should be able to apply a single transformation to the virtual camera pose instead of this constant query, right? If so, what's the transformation matrix?

Here's some copies of the errors I'm getting:

2023/11/09 06:17:40.493 18182 18205 Error Unity Error: GetUnityPose in the Magic Leap API failed. Reason: MLResult_PoseNotFound 
2023/11/09 06:17:40.493 18182 18205 Error Unity UnityEngine.XR.MagicLeap.MLResult:DidNativeCallSucceed(Code, String, Predicate`1, Boolean)
2023/11/09 06:17:40.493 18182 18205 Error Unity UnityEngine.XR.MagicLeap.MLAnchors:GetLocalizationInformation(LocalizationInfo&)
2023/11/09 06:17:40.493 18182 18205 Error Unity UnityEngine.XR.MagicLeap.MLAnchors:GetLocalizationInfo(LocalizationInfo&)
2023/11/09 06:17:40.493 18182 18205 Error Unity Argyle.Risa.AnchorTech.ML2.ML2RPManager:GetLocalizationInfo()
2023/11/09 06:17:40.493 18182 18205 Error Unity Argyle.RISA.AnchorTech.AnchorTechType:get_Current()
2023/11/09 06:17:40.493 18182 18205 Error Unity Argyle.RISA.<SelectRisaVersion>d__8:MoveNext()
2023/11/09 06:17:40.493 18182 18205 Error Unity System.Runtime.CompilerServices.AsyncVoidMethodBuilder:Start(TStateMachine&)
2023/11/09 06:17:40.493 18182 18205 Error Unity Argyle.RISA.RisaSelector:SelectRisaVersion()
2023/11/09 06:17:40.493 18182 18205 Error Unity 
2023/11/09 06:17:40.495 18182 18205 Error Unity Error: GetUnityPose in the Magic Leap API failed. Reason: MLResult_PoseNotFound 
2023/11/09 06:17:40.495 18182 18205 Error Unity UnityEngine.XR.MagicLeap.MLResult:DidNativeCallSucceed(Code, String, Predicate`1, Boolean)
2023/11/09 06:17:40.495 18182 18205 Error Unity UnityEngine.XR.MagicLeap.MLAnchors:GetLocalizationInformation(LocalizationInfo&)
2023/11/09 06:17:40.495 18182 18205 Error Unity UnityEngine.XR.MagicLeap.MLAnchors:GetLocalizationInfo(LocalizationInfo&)
2023/11/09 06:17:40.495 18182 18205 Error Unity Argyle.Risa.AnchorTech.ML2.ML2RPManager:GetLocalizationInfo()
2023/11/09 06:17:40.495 18182 18205 Error Unity Argyle.RISA.<SelectRisaVersion>d__8:MoveNext()
2023/11/09 06:17:40.495 18182 18205 Error Unity System.Runtime.CompilerServices.AsyncVoidMethodBuilder:Start(TStateMachine&)
2023/11/09 06:17:40.495 18182 18205 Error Unity Argyle.RISA.RisaSelector:SelectRisaVersion()
2023/11/09 06:17:40.495 18182 18205 Error Unity

kbabilinski · November 9, 2023, 6:44pm

Interesting, it looks like it's an issue coming from the Anchor manager rather than the camera it self. The Error in the anchors might be causing a frame hang which then causes the camera image to miss a frame as well. Do you get the Anchor localization issues when the camera get pose function is disabled?

I recomend migrating the GetLocalizationInfo from the Anchors API if you are calling it explicitly and using the MLSpaces API instead. It provides a callback when localization changes rather than requiring you to create the events yourself.

logan · November 9, 2023, 10:21pm

Oh I'm not using spaces. Not intentionally, anyway. It was too problematic. False positives, etc. The information I'm looking for here is really just the difference between the virtual and physical cameras pose.
EG If the virtual camera is centered on the left eye display and slightly converged toward a point in the line of the headset, then I would expect the physical camera offset to be a couple centimeters to the right and above. Then the rotation would be a couple degrees left of center and maybe a degree up. That would be a pose of something like

Vector3 positionOffset = new Vector3(.02f, .01f, 0f);
Vector3 eulerOffset = new Vector3(-1f, -2f, 0f);
			
Pose physicalPose = new Pose(
	virtualPose.position + positionOffset, 
	virtualPose.rotation * Quaternion.Euler(eulerOffset));

Everything else is already taken care of except knowing that precise offset.

kbabilinski · November 10, 2023, 6:31pm

Do you mind sharing the Argyle.Risa.AnchorTech.ML2.ML2RPManager:GetLocalizationInfo() function? It may be calling MLAnchors:GetLocalizationInfo() which could result in the error code. The error then could be delaying the GetFramePose function from the camera causing the Camera function to fail.

logan · November 21, 2023, 6:48am

I can share that function when I'm at my dev computer. It does seem to call that even though it shouldn't because I've cut off the use of Spaces. Must be left over from when I was trying to implement that tech.

For clarification, if I'm not using Magic Leap Spaces, should I even be using MLCVCamera.GetFramePose? My assumption would be that it is giving me the pose of the physical camera relative to Unity's global worldspace coordinate system. If it's giving relative to some OS level coordinate system then there are other transformations that need to take place.

And reiterating, if we can bypass this call by knowing the transformation matrix of virtual camera pose to physical camera pose, I believe that would be the most efficient and straightforward. I assume that transformation must be known. I can has?

kbabilinski · November 23, 2023, 4:43pm

Yes the MLCVCamera.GetFramePose function is the correct one to use to get the post of the camera given a timestamp. The pose is relative to the origin, which is where the headset began tracking head pose.

Since the frames are slightly delayed, using that function makes sure that they pose from the frame is exactly the pose of the camera.

logan · November 24, 2023, 6:07am

I’ve already accounted for delay. Does it also account for physical offset of the camera?

kbabilinski · November 24, 2023, 4:11pm

Yes, the Camera is calibrated at the factory so it will provide the most accurate pose. I strongly recommend continuing to use the GetFramePose call . If you were able to locate the previous MLAnchors:GetLocalizationInfo() logic and you are still having problems obtaining the CVCamera Pose please let me know. Have you noticed a performance hit when using this call?