ML2 OS version: Build: B3E.220619.07-R.124 MLSDK version: 0.53.1
When trying to get the CV Camera Pose, it seems like the rotation is valid, but the Position is behaving rather strangely.
Using the Camera pose to correct our rendering causes objects to shift away from the user as the head moves away from the origin of the world.
Replacing it with the Head pose gives significantly better results, albeit with a small offset, as expected from not using the exact source.
Any idea on what could be causing this behavior?
More details :
Camera pose is retrieved via MLCVCameraGetFramePose
Head pose is retrieved via MLSnapshotGetTransform
Here's a snippet of a few frames of lopgging of both positions given back from these calls
Head pose position : [-0.0185,-0.0148,0.1740]
Camera pose position : [-0.0285,0.0243,-0.0243]
Head pose position : [-0.0185,-0.0151,0.1735]
Camera pose position : [-0.0286,0.0241,-0.0244]
Head pose position : [-0.0190,-0.0157,0.1729]
Camera pose position : [-0.0287,0.0240,-0.0244]
Wanted to let you know that I am looking into this. While I track down a sample, it may be helpful if you provide a snippet of your code to show how you are obtaining the camera pose.
Thanks for sharing a sample Krystian, it sure helps validating everything
It seems we're doing the same calls as you are to get the camera position.
On Construction
result = (MLCVCameraTrackingCreate(&internal_data_->cv_camera_handle_));
On Processing a new camera frame
result = MLCVCameraGetFramePose(cv_camera_handle, head_handle, cv_camera_id, camera_timestamp_ns, &out_transform);
On Destruction
result = MLCVCameraTrackingDestroy(internal_data_->cv_camera_handle_);
One small difference I see, you update the pose of the camera as soon as a frame is available via on_video_buffer_available. We're currently caching the frame and processing it asap, this could be a few ms. Could that be an issue?
We're also getting the various handles on creation, not on Resume. It seems to work with everything else but could it cause unknown behaviors?
To make sure I understand. You're still suing the camera_timestamp from the captured frame but querying the location a few milliseconds later? You are also starting the camera capture On creation instead of overriding the application framework's on resume function?
The second part shouldn't matter but I look into if there are any known issues when ousing the camera_timestap later in the application.
MLCVCameraGetFramePos returns the location of the VCam with respect to world origin. Headpose is the location of RigDisplay (the average translation/rotation of the two displays) relative to world origin.
The VCam being slightly left, and higher makes sense but the large discrepancy in Z is strange. Do you have the s/n of the device so we can look at the lightwear.rig?
We've included a debug feature rendering the camera frustum on screen, using the camera position, it seems to remain very close to the world origin, whereas using the head position renders the frustum at the user position as expected. I could provide screenshots if needed.
I'm including a few more frames of log, with much bigger discrepancies, while moving around the origin in my room :
Head pose position : [0.0548,-0.0285,0.0350]
Camera pose position : [-0.0304,0.0215,-0.0246]
[.. a few frames later]
Head pose position : [0.1011,-0.0391,-0.0051]
Camera pose position : [-0.0309,0.0217,-0.0238]
Hi Stephane,
Are you getting any errors in the log when trying to read the camera pose. We are unable to reproduce the issue you are having using our test application. Are there any additional steps that we are missing?
I don't see anything obvious in the logs. No message when calling MLCVCameraGetFramePose
I do see some suspicious logs overal, all of them before we start tracking :
08-22 16:25:14.209 3342 10643 W ServiceManager: Permission failure: com.magicleap.permission.CAMERA_CAPTURE_CVIP from uid=10097 pid=12342
08-22 16:25:14.209 3342 10643 D ServiceManager: Permission check violation (permissive=0): com.magicleap.permission.CAMERA_CAPTURE_CVIP from uid=10097 pid=12342, Permission ARE enforcing.
We've tried enabling this permissions, calling system_helper_->CheckApplicationPermission seems to block the application.
We're currently running without it and are able to get access to frames.
This might be the only thing that we could change on our end.
I've also seen some suspicious system logs, maybe they could help you out :
08-22 16:25:15.539 3274 10529 E android.hardware.camera.provider.mero@2.4-service_64: Failed to get IAshmemDeviceService.
[..]
08-22 16:34:08.431 3342 9349 E Camera3-Device: Camera 1: configureStreamsLocked: Set of requested inputs/outputs not supported by HAL
08-22 16:34:08.431 3342 9349 E CameraDeviceClient: endConfigure: Camera 1: Unsupported set of inputs/outputs provided
08-22 16:34:08.431 14331 14353 E Camera-Device: endConfigure fail Status(-8, EX_SERVICE_SPECIFIC): '3: endConfigure:733: Camera 1: Unsupported set of inputs/outputs provided'
[..]
08-22 16:34:08.438 3274 3274 W amdCamera3VirtualCamera: Can not find zsl stream Id for camera (1)
This has been developped for the ML1, we've now modified our code to call MLCVCameraGetFramePose() during the on_video_buffer_available() callback directly and this is now providing accurate results!
I guess the data gets invalidated on your end when returning from the callback?
