Error messages from logs (syntax-highlighting is supported via Markdown):
I'm using camera feed for precise CV work where I need very correct locations of pixels for ray casting etc. So I'm following Intrinsic/Extrinsic Parameters | MagicLeap Developer Documentation to get the info to transform the results. The distortion correction seems to be doing its job (tough to be 100% sure) but when I try to get the camerapose, I hit catastrophic errors.
By way of example, when I uncomment in the following code, suddenly my app won't keep rendering past a few frames
private void SetCapturePose(MLCameraBase.ResultExtras resultextras)
{
Debug.Log($"Capture: Starting pose finder. \n" +
$"Timestamp: {resultextras.VCamTimestamp} \n" +
$"Intrinsics: {resultextras.Intrinsics?.ToString() ?? "null"}");
//Set pose info for later use in correction and raycasting.
Transform camTform = Cam.transform;
var cameraPose = new Pose(camTform.position, camTform.rotation);
// if (MLCVCamera.GetFramePose(resultextras.VCamTimestamp, out Matrix4x4 cameraTransform).IsOk)
// CapturePose = new Pose(cameraTransform.GetPosition(), cameraTransform.rotation);
// else
// Debug.LogError("Capture: Attempting to get pose from resultExtras failed: VCamTimestamp is 0");
Debug.Log($"Camera pose: {cameraPose} \n" +
$"Capture pose: {CapturePose}");
}
I'm wondering if the sample code is missing context for how we need to get at that information. It also seems like the informaiton I'm looking for (the pose of the physical camera) should have a static relationship to the virtual camera so I should be able to apply a single transformation to the virtual camera pose instead of this constant query, right? If so, what's the transformation matrix?
Interesting, it looks like it's an issue coming from the Anchor manager rather than the camera it self. The Error in the anchors might be causing a frame hang which then causes the camera image to miss a frame as well. Do you get the Anchor localization issues when the camera get pose function is disabled?
I recomend migrating the GetLocalizationInfo from the Anchors API if you are calling it explicitly and using the MLSpaces API instead. It provides a callback when localization changes rather than requiring you to create the events yourself.
Oh I'm not using spaces. Not intentionally, anyway. It was too problematic. False positives, etc. The information I'm looking for here is really just the difference between the virtual and physical cameras pose.
EG If the virtual camera is centered on the left eye display and slightly converged toward a point in the line of the headset, then I would expect the physical camera offset to be a couple centimeters to the right and above. Then the rotation would be a couple degrees left of center and maybe a degree up. That would be a pose of something like
Vector3 positionOffset = new Vector3(.02f, .01f, 0f);
Vector3 eulerOffset = new Vector3(-1f, -2f, 0f);
Pose physicalPose = new Pose(
virtualPose.position + positionOffset,
virtualPose.rotation * Quaternion.Euler(eulerOffset));
Everything else is already taken care of except knowing that precise offset.
Do you mind sharing the Argyle.Risa.AnchorTech.ML2.ML2RPManager:GetLocalizationInfo() function? It may be calling MLAnchors:GetLocalizationInfo() which could result in the error code. The error then could be delaying the GetFramePose function from the camera causing the Camera function to fail.
I can share that function when I'm at my dev computer. It does seem to call that even though it shouldn't because I've cut off the use of Spaces. Must be left over from when I was trying to implement that tech.
For clarification, if I'm not using Magic Leap Spaces, should I even be using MLCVCamera.GetFramePose? My assumption would be that it is giving me the pose of the physical camera relative to Unity's global worldspace coordinate system. If it's giving relative to some OS level coordinate system then there are other transformations that need to take place.
And reiterating, if we can bypass this call by knowing the transformation matrix of virtual camera pose to physical camera pose, I believe that would be the most efficient and straightforward. I assume that transformation must be known. I can has?
Yes the MLCVCamera.GetFramePose function is the correct one to use to get the post of the camera given a timestamp. The pose is relative to the origin, which is where the headset began tracking head pose.
Since the frames are slightly delayed, using that function makes sure that they pose from the frame is exactly the pose of the camera.
Yes, the Camera is calibrated at the factory so it will provide the most accurate pose. I strongly recommend continuing to use the GetFramePose call . If you were able to locate the previous MLAnchors:GetLocalizationInfo() logic and you are still having problems obtaining the CVCamera Pose please let me know. Have you noticed a performance hit when using this call?
Update: I was able to start getting the pose by reducing the frequency of query. If I save out resultExtras.VCamTimestamp each time I capture an image, then when I want to do CV on a particular image, I do
MLCVCamera.GetFramePose(_lastCaptureTimestamp, out Matrix4x4 cameraTransform)
to get the pose that matches the image I'm processing.
I've been able to get results doing that. So maybe it was tripping over itself trying to grab pose at 30 fps?
Thank you for the update. That would make sense, when you were getting the pose every frame, were you processing it using the customer marker tracking library at 30fps as well? I assume that the reduced frequency freed up resources preventing your device from rending the blank screen.
Based on your message it seems like the issue is resolved / you were able to find a workaround?
Nevertheless, we are working on tracking down why the MLCVCamera.GetFramePose call results in the screen going blank when performing resource intensive tasks, as this is not the intended behavior . However I want to make sure you have a valid workaround while we work on the bug.
I have that solution to the pose problem and it's working well. But as I attempt to close up this effort and move forward, I'm having possibly related issues with capture itself. I started a new thread for it: