Camera API Compatibility with OpenXR, OS and Vuforia

zhhqu0305 · August 15, 2024, 12:08am

Give us as much detail as possible regarding the issue you're experiencing:

Unity Editor version: 2022.3.6f1
ML2 OS version: 1.8.0
Unity SDK version: 2.3.0
Host OS: Windows

I was working on updating an existing project to newer versions of MLSDK. Having some trouble with trying to obtain an RGB camera image, I found this post and tried to revert my project. However, the following bug happened, when I tried to determine the camera pose in the callback with if (MLCVCamera.GetFramePose(extras.VCamTimestamp, out Matrix4x4 cameraTransform).IsOk):

08-14 18:56:22.954  1047  3270  3343 E amdCamera3MetaArb:  mAFMode : 4
08-14 18:56:22.954  1047  3270  3343 E amdCamera3MetaArb:  ANDROID_CONTROL_AF_TRIGGER : 0
08-14 18:56:22.954  1047  3270  3343 E amdCamera3MetaArb:  mAWBMode : 1
08-14 18:56:22.988  1047  3270  3343 E amdCamera3MetaArb:  mAFMode : 4
08-14 18:56:22.988  1047  3270  3343 E amdCamera3MetaArb:  ANDROID_CONTROL_AF_TRIGGER : 0
08-14 18:56:22.988  1047  3270  3343 E amdCamera3MetaArb:  mAWBMode : 1
08-14 18:56:23.021  1047  3270  3343 E amdCamera3MetaArb:  mAFMode : 4
08-14 18:56:23.021  1047  3270  3343 E amdCamera3MetaArb:  ANDROID_CONTROL_AF_TRIGGER : 0
08-14 18:56:23.021  1047  3270  3343 E amdCamera3MetaArb:  mAWBMode : 1
08-14 18:56:23.015 10107  3510  3510 W Binder:3510_4: type=1400 audit(0.0:79031): avc: denied { search } for name="traces" dev="nvme0n1p37" ino=5316611 scontext=u:r:untrusted_app:s0:c107,c256,c512,c768 tcontext=u:object_r:trace_data_file:s0 tclass=dir permissive=0
08-14 18:56:23.055  1047  3270  3343 E amdCamera3MetaArb:  mAFMode : 4
08-14 18:56:23.055  1047  3270  3343 E amdCamera3MetaArb:  ANDROID_CONTROL_AF_TRIGGER : 0
08-14 18:56:23.055  1047  3270  3343 E amdCamera3MetaArb:  mAWBMode : 1
08-14 18:56:23.088  1047  3270  3343 E amdCamera3MetaArb:  mAFMode : 4
08-14 18:56:23.088  1047  3270  3343 E amdCamera3MetaArb:  ANDROID_CONTROL_AF_TRIGGER : 0
08-14 18:56:23.088  1047  3270  3343 E amdCamera3MetaArb:  mAWBMode : 1
08-14 18:56:23.121 10107  3510  3560 E BufferQueueConsumer: [BufferItemQueue] acquireBuffer: max acquired buffer count reached: 5 (max 4)
08-14 18:56:23.121 10107  3510  3560 E BufferItemConsumer: [BufferItemQueue] Error acquiring buffer: Unknown error -38 (-38)
08-14 18:56:23.121 10107  3510  3560 E         : acquireBuffer BufferItemConsumer::acquireBuffer fail, status: -38
08-14 18:56:23.121  1047  3270  3343 E amdCamera3MetaArb:  mAFMode : 4
08-14 18:56:23.121  1047  3270  3343 E amdCamera3MetaArb:  ANDROID_CONTROL_AF_TRIGGER : 0
08-14 18:56:23.121  1047  3270  3343 E amdCamera3MetaArb:  mAWBMode : 1
08-14 18:56:23.154 10107  3510  3560 E BufferQueueConsumer: [BufferItemQueue] acquireBuffer: max acquired buffer count reached: 5 (max 4)
08-14 18:56:23.154 10107  3510  3560 E BufferItemConsumer: [BufferItemQueue] Error acquiring buffer: Unknown error -38 (-38)
08-14 18:56:23.154 10107  3510  3560 E         : acquireBuffer BufferItemConsumer::acquireBuffer fail, status: -38
08-14 18:56:23.154  1047  3270  3343 E amdCamera3MetaArb:  mAFMode : 4
08-14 18:56:23.154  1047  3270  3343 E amdCamera3MetaArb:  ANDROID_CONTROL_AF_TRIGGER : 0
08-14 18:56:23.154  1047  3270  3343 E amdCamera3MetaArb:  mAWBMode : 1
08-14 18:56:23.188 10107  3510  3560 E BufferQueueConsumer: [BufferItemQueue] acquireBuffer: max acquired buffer count reached: 5 (max 4)
08-14 18:56:23.188 10107  3510  3560 E BufferItemConsumer: [BufferItemQueue] Error acquiring buffer: Unknown error -38 (-38)
08-14 18:56:23.188 10107  3510  3560 E         : acquireBuffer BufferItemConsumer::acquireBuffer fail, status: -38
08-14 18:56:23.221 10107  3510  3560 E BufferQueueConsumer: [BufferItemQueue] acquireBuffer: max acquired buffer count reached: 5 (max 4)
08-14 18:56:23.221 10107  3510  3560 E BufferItemConsumer: [BufferItemQueue] Error acquiring buffer: Unknown error -38 (-38)
08-14 18:56:23.221 10107  3510  3560 E         : acquireBuffer BufferItemConsumer::acquireBuffer fail, status: -38
08-14 18:56:23.254 10107  3510  3560 E BufferQueueConsumer: [BufferItemQueue] acquireBuffer: max acquired buffer count reached: 5 (max 4)
08-14 18:56:23.254 10107  3510  3560 E BufferItemConsumer: [BufferItemQueue] Error acquiring buffer: Unknown error -38 (-38)
08-14 18:56:23.254 10107  3510  3560 E         : acquireBuffer BufferItemConsumer::acquireBuffer fail, status: -38
08-14 18:56:23.620  4045  4364  4364 D BluetoothGatt: readRssi() - device: C4:D0:93:3C:22:33
08-14 18:56:24.254  1047  3367  3912 E Camera3-OutputStream: getBufferLockedCommon: Stream 0: Can't dequeue next output buffer: Connection timed out (-110)
08-14 18:56:24.254 10107  3510  3856 W Camera-Device: onDeviceError is invoked with error code: 3 frame number = 111
08-14 18:56:24.255 10107  3510  3982 I ml_camera_client: OnFailed
08-14 18:56:24.621  4045  4364  4364 D BluetoothGatt: readRssi() - device: C4:D0:93:3C:22:33
08-14 18:56:24.691  4116  5282  5283 I vcamdmmcontroller_service: nova/frameworks/services/perception_services/vcamdmmcontroller_svc/bin/src/main.cpp(182) ListenOnZpad():
08-14 18:56:24.691  4116  5282  5283 I vcamdmmcontroller_service: INF: ZPAD PW_MS_CONTROL_ZPAD still waiting...
08-14 18:56:24.904 10107  3510  3563 E com.UnityTechnologies.com.unity.template.urpblank: leapcore/frameworks/perception/data_sources/include/pad/xpad_data_source.h(105) GetClosestTimestampedData():
08-14 18:56:24.904 10107  3510  3563 E com.UnityTechnologies.com.unity.template.urpblank: ERR: Data Not Found for timestamp: 20862039321us, now time: 20864876890us
08-14 18:56:24.905 10107  3510  3563 E Unity   : Error: MLCVCameraGetFramePose in the Magic Leap API failed. Reason: MLResult_PoseNotFound 
08-14 18:56:24.905 10107  3510  3563 E Unity   : Error: MLCamera.InternalGetFramePose failed to get camera frame pose. Reason: MLResult_PoseNotFound
08-14 18:56:24.905 10107  3510  3563 I Unity   : Could not get frame pose
08-14 18:56:24.899 10107  3510  3510 W UnityGfxDeviceW: type=1400 audit(0.0:79041): avc: denied { search } for name="traces" dev="nvme0n1p37" ino=5316611 scontext=u:r:untrusted_app:s0:c107,c256,c512,c768 tcontext=u:object_r:trace_data_file:s0 tclass=dir permissive=0
08-14 18:56:24.899 10107  3510  3510 W UnityGfxDeviceW: type=1400 audit(0.0:79042): avc: denied { search } for name="traces" dev="nvme0n1p37" ino=5316611 scontext=u:r:untrusted_app:s0:c107,c256,c512,c768 tcontext=u:object_r:trace_data_file:s0 tclass=dir permissive=0
08-14 18:56:25.042  1000  3293  3293 I MagicFlinger: INF: MagicFlinger Periodic Info Log: CompositionCPU=0.65ms ; CompositionGPU=1.35ms ; ClientCPU=4.09ms ; ClientGPU=5.77ms ; TotalCPU=4.74ms ; TotalGPU=7.13ms ; ClientDroppedFrames=210 ; GlobalDimmer=0%
08-14 18:56:25.255  1047  3367  3912 E Camera3-OutputStream: getBufferLockedCommon: Stream 0: Can't dequeue next output buffer: Connection timed out (-110)
08-14 18:56:25.255 10107  3510  3885 W Camera-Device: onDeviceError is invoked with error code: 3 frame number = 112
08-14 18:56:25.255 10107  3510  3885 W Camera-Device: Error 3 has been posted, skip error 3 notification
08-14 18:56:25.621  4045  4364  4364 D BluetoothGatt: readRssi() - device: C4:D0:93:3C:22:33
08-14 18:56:26.255  1047  3367  3912 E Camera3-OutputStream: getBufferLockedCommon: Stream 0: Can't dequeue next output buffer: Connection timed out (-110)
08-14 18:56:26.255 10107  3510  3856 W Camera-Device: onDeviceError is invoked with error code: 3 frame number = 113
08-14 18:56:26.255 10107  3510  3856 W Camera-Device: Error 3 has been posted, skip error 3 notification
08-14 18:56:26.373 10107  3510  3925 E com.UnityTechnologies.com.unity.template.urpblank: leapcore/frameworks/perception/data_sources/include/pad/xpad_data_source.h(105) GetClosestTimestampedData():
08-14 18:56:26.373 10107  3510  3925 E com.UnityTechnologies.com.unity.template.urpblank: ERR: Data Not Found for timestamp: 20862893213us, now time: 20866345487us
08-14 18:56:26.377 10107  3510  3563 E com.UnityTechnologies.com.unity.template.urpblank: leapcore/frameworks/perception/data_sources/include/pad/xpad_data_source.h(105) GetClosestTimestampedData():
08-14 18:56:26.377 10107  3510  3563 E com.UnityTechnologies.com.unity.template.urpblank: ERR: Data Not Found for timestamp: 20864881193us, now time: 20866349415us
08-14 18:56:26.377 10107  3510  3563 E com.UnityTechnologies.com.unity.template.urpblank: leapcore/frameworks/perception/data_sources/include/pad/xpad_data_source.h(105) GetClosestTimestampedData():
08-14 18:56:26.377 10107  3510  3563 E com.UnityTechnologies.com.unity.template.urpblank: ERR: Data Not Found for timestamp: 20864881193us, now time: 20866349524us

It appears that after updating to the newer OS version, this query starts to be too slow (while it worked on OS 1.4.1). I tried to use the Pixel Sensor's Get Pose API however it also says:

08-14 19:47:34.873 10107 18542 18562 E Unity   :  was not connected
08-14 19:47:34.873 10107 18542 18562 E Unity   : MagicLeap.OpenXR.Features.PixelSensors.MagicLeapPixelSensorFeature:IsSensorConnected(PixelSensorId, PixelSensor&, Boolean)
08-14 19:47:34.873 10107 18542 18562 E Unity   : MagicLeap.OpenXR.Features.PixelSensors.MagicLeapPixelSensorFeature:GetSensorPose(PixelSensorId, Pose)
08-14 19:47:34.873 10107 18542 18562 E Unity   : SimpleCamera:GetSensorPose()
08-14 19:47:34.873 10107 18542 18562 E Unity   : SimpleCamera:RawImageAvailable(CameraOutput, ResultExtras, Metadata)
08-14 19:47:34.873 10107 18542 18562 E Unity   : UnityEngine.XR.MagicLeap.Native.DispatchPayload3`3:Dispatch()
08-14 19:47:34.873 10107 18542 18562 E Unity   : UnityEngine.XR.MagicLeap.Native.MLThreadDispatch:DispatchAll()
08-14 19:47:34.873 10107 18542 18562 E Unity   : 
08-14 19:47:34.873 10107 18542 18562 I Unity   : Sensor Pose:((0.00, 0.00, 0.00), (0.00000, 0.00000, 0.00000, 0.00000))

I setup the project with setup tool 2.0.12, and when prompted to choose between OpenXR and ML SDK, I chose ML SDK but also enabled OpenXR in Project Settings - XR Plug-in Management later. It now looks like this:

My guess is that the usage of Vuforia somehow led to both issues, (1) that OpenXR cannot find it when Vuforia claims the camera and (2) Vuforia made it too slow for the MLCamera API to return the pose. Is there a possible fix to this issue?

I remember having some issues with the Vuforia CameraDevice captures, particularly in the aspect ratio as I mentioned here. I didn't fix that as I moved to the MLCamera implementation back then. Is it possible to obtain RGB camera access using the Vuforia API? It might be a temporary workaround before you figure out a way to integrate camera access with the new OpenXR workflow. With the camera pose available from the Pixel Sensor's Get Pose API (or maybe through MRTK?), maybe it's still possible to do a world-to-screen conversion? I'm trying to work with OpenXR + MRTK3 + Vuforia as suggested here, but would really appreciate any help on this.

Thanks in advance!

aheaney · August 15, 2024, 6:24pm

Hello,

Sorry about the trouble. Thanks for including the logs. Based on the logs, it looks like the app was attempting to request poses that are too old (1.5-3 seconds looking at the log). Let's see if we can start to address these issues-

Using ML SDK-

In order for the GetFramePose api to work when using the OpenXR provider, I believe you need to enable perception snapshots. You can enable it from the Project Settings -> XR Plugin Management -> OpenXR -> Gear icon next to MagicLeap 2 support. You likely already did this if you got this far in an OpenXR project, but just to check-

How were you getting the timestamp using ml sdk? Were you calling GetFramePose in a camera capture callback? In Unity that callback would need to be raised on Unity's update thread, so there's probably inherent potential for time loss, but 1.5s+ is quite a lot. How is the performance of your application generally? As a sanity check, it might be worth adding logging as early as possible when you get the callback and after GetFramePose returns. I imagine that if your app is useable, then app script code isn't blocking for > 1s. Would you be able to grab a full bugreport immediately after reproducing the issue using the CV camera apis? ADB Bugreport | MagicLeap Developer Documentation
The bug report captures full logcat logs (using a ring buffer, it's not infinite) and other information. We could take a look at it. Feel free to dm it to me as that might be a bit much to post publicly.

Using pixel sensor-

As for using the pixel sensor api, when do you call CreatePixelSensor? The call to create the sensor need only be called once and you can do it well ahead of calls to GetSensorPose.

Using Vuforia certainly adds a wrinkle. Not sure how feasible it would be to effectively disable any usage of it in your project. I don't know that it should prevent access to the camera. If your code that uses the pixel sensor apis is reasonably isolated, would you be able to post a snip showing how you initialize the sensor and attempt to get the sensor pose? Maybe we can repro.

Best,
Adam

zhhqu0305 · August 15, 2024, 7:53pm

Hi Adam,

Thank you for your response! It seems that I did not enable perception snapshots before - but even after enabling it, the results did not change as much. I do want to note that, I really haven't been playing with OpenXR a lot - Iast time I tried this was at OS 1.6.0, when the same bug happened (but at that time I think there's also a bug in the timestamp) that forced me to revert back.

I'm sharing the code here for your reference. In short I tried to capture an image every 1.5 seconds, and need the camera pose to convert 4 world coordinates to screen coordinates to crop the image and send to a server. This code ran okay on OS 1.4.0-dev1 (well, it's only "okay" since it's really unstable, sometimes it ran very smoothly, sometimes there will be observable stucks when the capture happens, and always after running it for several months, the app crashes very often with the frame rate starts to drop under 30, and I will run a factory reset and re-build the same project to solve this), but starting from 1.6.0 the camera code I wrote simply stopped working. In this iteration I started to experiment with pixel sensor but tbh I have no idea if I'm doing things correctly. I'll also dm you the bug report - FYI, I always start by searching for "image capture" in the file, if this can make your job easier. I have Vuforia tracking 5 image targets at the same time, but (1) 3 image targets also led to timeout and (2) 5 worked fine on OS 1.4.0-dev1.

Also, not sure if this matters, but I'm not using the XR Rig since the Vuforia setup guide asks for the Magic Leap Main Camera Prefab. The code was attached to that main camera.

SimpleCameraTest.cs (11.0 KB)

Looking forward to hearing back from you!

Best,
Zhehan

zhhqu0305 · August 15, 2024, 7:56pm

Actually, how can I dm you? I was in a conversation with somebody before but don't know how to initiate one.

aheaney · August 16, 2024, 12:36am

Hi Zhehan,

Thanks for getting back. You should be able to send me a message by clicking on my username and then there should be a message button that appears.

It sounds like there generally may be some performance issues with the app. Target frame rate should be 60 fps ideally.

Taking a quick look at the code sample, I didn't see a call to pixelSensorFeature.CreatePixelSensor(sensorType). I think you still need to at least call that to initialize the sensor even though you're only using it to access pose data.

Note that sensorType is a complex type. You can use pixelSensorFeature to query for the list of sensors and pass the one that corresponds to the rgb camera, something like this- pixelSensorFeature.GetSupportedSensors().Find(x => x.SensorName.Contains("Picture Center"))
Also, I saw that you get the sensor type by calling pixelSensorFeature.GetSupportedSensors().FirstOrDefault(). I think that the rgb cam happens to be the first one in the list, but you might want to check.

As for the issues with ML SDK, we could file feedback internally if there is clear performance degradation across OS versions. Are you able to test using less image targets with Vuforia? What happens if you only use 1? Btw, I wouldn't anticipate running a factory reset on the device or rebuilding the project really making a difference. If anything, you might try simply killing and restarting your app or uninstalling/reinstalling it if it persists anything.

Anyways, handful of observations and things you can try-

It looks like the sample code you sent is trying to use both apis (CV camera & pixel sensor) at once. I'd only test one at a time.

In the RawImageAvailable callback handler, I noticed a call to your GetSensorPose() method, which uses the pixel sensor api, as the only thing that happens before the call to GetFramePose. Could you remove that call and let GetFramePose happen immediately?

Also, it looks like you're setting the capture frame rate to 60fps. I believe that you can set that to 'None' as your just capturing stills and pumping the capture with your own loop. 60 is rather fast for image capture. I think it would be more appropriate for the video/preview capture modes, but I'm not sure if it's harmful for Image capture. Since you're pumping it anyways though (at a much lower rate), you might just set it to None.

Something else to look into-
MLCamera does have async apis- see CreateAndConnectAsync, CaptureImageAsync, but I don't think CaptureImage() is blocking since it relies on a callback anyways

Best,
Adam

zhhqu0305 · August 16, 2024, 12:59am

Hi Adam,

Thanks for your detailed feedback! I'll try what you've suggested tomorrow, and send you back a bug report. I still cannot figure out how to send a message to you - see the attached screenshot when I click on your username.

While it sounds not practical, factory reset did solve my issue, like at least three times in the past (see this that I posted). I think it might be related with this post, but I haven't got a chance to test it with the latest sdks. Will do this soon.

Thanks!

aheaney · August 16, 2024, 3:48am

Hm okey sorry, I might have to message you first. I just didn't want you to have to post a bugreport publicly. Anyways, sounds good. Btw, wrt to my last comment, as I recall there were some potential performance issues with the MLCamera apis- in particular if multiple cameras were used simultaneously. It's possible that things might actually improve if you use the Async version of the capture api. (which does the same thing, it'll just make call the C api call off of Unity's update thread)

zhhqu0305 · August 19, 2024, 6:46pm

Update: I have tried to use the OpenXR + MRTK3 + Vuforia pipe, but also encountered bugs that stopped me from going forward.

The Vuforia Digital Eyewear sample uses Unity package 2.1.0 and Magic Leap MRTK 1.0.0, but PixelSensor was only supported since 2.2.0.
Directly importing the vuforia sample to a 2.3.0 project (I set it up following this link) leads to code compilation errors.
Adding only the assets (prefabs, including the MRTK Camera, see figure below) and the Vuforia package 10.25.4 to the project configured with MRTK3, the project can build after some simple code fixes, but the same timestamp bug appears after built onto Magic Leap 2 (see log below).

08-19 13:57:38.819 10107 14674 14692 E BufferQueueConsumer: [BufferItemQueue] acquireBuffer: max acquired buffer count reached: 5 (max 4)
08-19 13:57:38.819 10107 14674 14692 E BufferItemConsumer: [BufferItemQueue] Error acquiring buffer: Unknown error -38 (-38)
08-19 13:57:38.819 10107 14674 14692 E         : acquireBuffer BufferItemConsumer::acquireBuffer fail, status: -38
08-19 13:57:38.852 10107 14674 14692 E BufferQueueConsumer: [BufferItemQueue] acquireBuffer: max acquired buffer count reached: 5 (max 4)
08-19 13:57:38.852 10107 14674 14692 E BufferItemConsumer: [BufferItemQueue] Error acquiring buffer: Unknown error -38 (-38)
08-19 13:57:38.852 10107 14674 14692 E         : acquireBuffer BufferItemConsumer::acquireBuffer fail, status: -38
08-19 13:57:38.886 10107 14674 14690 E BufferQueueConsumer: [BufferItemQueue] acquireBuffer: max acquired buffer count reached: 5 (max 4)
08-19 13:57:38.886 10107 14674 14690 E BufferItemConsumer: [BufferItemQueue] Error acquiring buffer: Unknown error -38 (-38)
08-19 13:57:38.886 10107 14674 14690 E         : acquireBuffer BufferItemConsumer::acquireBuffer fail, status: -38
08-19 13:57:38.912  4045  4275  4275 D BluetoothGatt: readRssi() - device: C4:D0:93:3C:22:33
08-19 13:57:38.919 10107 14674 14692 E BufferQueueConsumer: [BufferItemQueue] acquireBuffer: max acquired buffer count reached: 5 (max 4)
08-19 13:57:38.919 10107 14674 14692 E BufferItemConsumer: [BufferItemQueue] Error acquiring buffer: Unknown error -38 (-38)
08-19 13:57:38.919 10107 14674 14692 E         : acquireBuffer BufferItemConsumer::acquireBuffer fail, status: -38
08-19 13:57:38.952 10107 14674 14692 E BufferQueueConsumer: [BufferItemQueue] acquireBuffer: max acquired buffer count reached: 5 (max 4)
08-19 13:57:38.952 10107 14674 14692 E BufferItemConsumer: [BufferItemQueue] Error acquiring buffer: Unknown error -38 (-38)
08-19 13:57:38.952 10107 14674 14692 E         : acquireBuffer BufferItemConsumer::acquireBuffer fail, status: -38
08-19 13:57:39.912  4045  4275  4275 D BluetoothGatt: readRssi() - device: C4:D0:93:3C:22:33
08-19 13:57:39.952  1047  3369 14883 E Camera3-OutputStream: getBufferLockedCommon: Stream 0: Can't dequeue next output buffer: Connection timed out (-110)
08-19 13:57:39.953 10107 14674 14806 W Camera-Device: onDeviceError is invoked with error code: 3 frame number = 41
08-19 13:57:39.953 10107 14674 14909 I ml_camera_client: OnFailed
08-19 13:57:40.912  4045  4275  4275 D BluetoothGatt: readRssi() - device: C4:D0:93:3C:22:33
08-19 13:57:40.953  1047  3369 14883 E Camera3-OutputStream: getBufferLockedCommon: Stream 0: Can't dequeue next output buffer: Connection timed out (-110)
08-19 13:57:40.953 10107 14674 14692 W Camera-Device: onDeviceError is invoked with error code: 3 frame number = 42
08-19 13:57:40.953 10107 14674 14692 W Camera-Device: Error 3 has been posted, skip error 3 notification
08-19 13:57:41.028  4116  5245  5248 I vcamdmmcontroller_service: nova/frameworks/services/perception_services/vcamdmmcontroller_svc/bin/src/main.cpp(182) ListenOnZpad():
08-19 13:57:41.028  4116  5245  5248 I vcamdmmcontroller_service: INF: ZPAD PW_CONFIG_MAPPING_MODE_ZPAD still waiting...
08-19 13:57:41.087 10107 14674 14695 E com.UnityTechnologies.com.unity.template.urpblank: leapcore/frameworks/perception/data_sources/include/pad/xpad_data_source.h(105) GetClosestTimestampedData():
08-19 13:57:41.087 10107 14674 14695 E com.UnityTechnologies.com.unity.template.urpblank: ERR: Data Not Found for timestamp: 10650905302us, now time: 10653925414us

If you want to see the bug report as well, I can share it with you personally, but it seems to suggest that the Vuforia sample did something special to prevent this from happening, which is not available in the ML Unity package 2.2.0+. I can also reach out to the Vuforia side about this, but from your perspective is there anything I can potentially do here to make things work?

Best,
Zhehan

kbabilinski · August 19, 2024, 8:55pm

Were you able to get the Vuforia sample to work in the previous project you were working on?

Did you enable the Perception Snapshots setting inside the Magic Leap 2 Open XR Support Feature in the new project?

What type of code compilation errors do you see when importing the sample into the new project?

zhhqu0305 · August 19, 2024, 9:48pm

Hi,

Thanks for your reply.

In the previous project, Vuforia is working but the camera GetFramePose() is not; Adding the new digitial eyewear sample requires a whole new setup.
Not sure what you mean by the "new project" - if you're talking about the projects without relying on MRTK3, yes I did enable it; if it's about the ones with MRTK3 integrated, not yet as I haven't really figured out a way to work on Unity sdk >2.2.0
When importing, I think the error is related with namespaces (I think it's related with using UnityEngine.XR.OpenXR.Features.MagicLeapSupport). I also remember having more bugs after I resolved this by chance, but I cannot reproduce this in a short time as I already updated the project. Should be easy if you would like to reproduce - simply set up the project with unity sdk 2.3.0 and MRTK3 magic leap package 1.1.0 and then import the sample project.

kbabilinski · August 19, 2024, 11:36pm

Thank you for that information. Just to recap the issue:

The first project used to "work" on the previous version of the OS but now returns an error regarding the timestamp. This project uses the MLSDK and not OpenXR.

When you run the application on the new version of the OS does the Pose fail immediately or does it appear to obtain the initial pose before failing? Updating the code to use the async capture method did not resolve the issue?

If it fails immediately it might be due to a change that was introduced in OS version 1.6.0. I'm not sure which version of the Magic Leap OS the Vuforia Plugin targets, but I do know that in version 1.6.0 the ML Camera API was updated to return nanoseconds instead of microseconds this change could cause the frame pose to return null if the plugin was not updated.

Understood
The Unity SDK was updated and some of the namespaces were removed to better align with other platforms. Here is a description of the namespace migration and a tool that can help you migrate the namespaces in your file: [Unity SDK 2.3.0] Unity Namespace Changes - #4

zhhqu0305 · August 20, 2024, 12:08am

Thanks for your reply!

I am aware of the nanosecond change but I don't think it's causing the trouble - I manually did the conversion in 1.6.0, and in 2.3.0 I think this has been fixed as in the log it reads

08-16 14:29:59.581946 23510 23532 E com.UnityTechnologies.com.unity.template.urpblank: ERR: Data Not Found for timestamp: 4648243823us, now time: 4649570314us

On the other hand, I think the flipVertical method was only fixed since 2.2.0, so I chose to use 2.3.0 for testing this. It did not give me the initial pose (at least from what I can tell), but I haven't tried to do it the async way (not sure how to call an async method at a fixed interval).

Thanks for point to the resource. I'll try it as soons as possible - but at the same time I think importing the Vuforia Digital Eyewear Sample forces to import Unity SDK 2.1.0, so I don't know how this would work out. I remember it threw an error about this when I import, but after I manually changed the names and then the editor restarts, after which I have to change it back so that it follows the Unity SDK 2.1.0 convention. I don't think there would be an easy way to integrate Vuforia before the release of a new sample, or a setup tutorial?

kbabilinski · August 20, 2024, 12:36am

My recommendation would be to do the following :

1.Get Vuforia working inside a Unity Project with the Magic Leap SDK.

Once you have Vuforia working, use the Magic Leap Camera Example that is on the Magic Leap Developer portal. The example script will provide a callback when a video frame is retrieved. (If you use the OpenXR workflow enable the perception snapshots setting)
Use the camera visualization example posted on the developer docs to visualize the frame inside your app.
Build and run to make sure it works as expected.
Add the Get frame pose function and log the camera position.
Run the example to see if it works as expected.
Integrate both Vuforia and the camera script to verify that you are able to obtain the pose.
Finally add your additional image processing logic

Let me know if you run into any issues or have any questions.

zhhqu0305 · August 20, 2024, 12:49am

Sounds good - are you suggesting that, at this moment, I shouldn't try to work with the OpenXR functions?

kbabilinski · August 20, 2024, 12:51am

You can use OpenXR, the suggestion is to incrementally add the functionality to better understand where the issue occurs so we can troubleshoot effectively.

zhhqu0305 · August 20, 2024, 5:12pm

Following the steps you provided, what I encountered was:

(1) when I import the setup tool, I am prompted to choose between "OpenXR" and "Magic Leap sdk (deprecated)". I'm not sure how it works - if I select "OpenXR", the "Magic Leap" provider would disappear from the list of android providers, and I won't be able to complete the steps as specified here. If I proceed with this setup to add a Vuforia target, the app crashes as soon as it launches.

(2) If I select "Magic Leap sdk (deprecated)", I can proceed and integrate Vuforia. The Camera code worked well, and using the Video callback I can successfully query CameraFramePose. However, I can see the application being laggy from time to time when running the code to capture video at 1440*1080, 30FPS, and if I switch back to my old way of capturing one image per 1.5 seconds (code attached), the framepose cannot be returned. (see attached log, it seems to be slightly exceeding 500ms before I add any additional logic). Is there a reason that capturing one image per 1.5s makes this worse? What would you suggest I do in this case?

Best,
Zhehan

NewSimpleCamera.cs (6.0 KB)

G472XT10013B_device-log.txt (691.7 KB)

kbabilinski · August 20, 2024, 6:31pm

The issue you are running into is that the

MLResult result = _camera.CaptureImage(1);

Hangs the main thread which causes the delay. You can use the async method instead and before capturing the next image:

 // Use the PrepareCapture method to set the capture configuration and get the metadata handle
MLResult prepareCaptureResult = colorCamera.PrepareCapture(captureConfig, out MLCamera.Metadata _);

 var result = await colorCamera.CaptureImageAsync();
            if (!result.IsOk)
            {
                Debug.LogError("Image capture failed!");
            }
            else
            {
                // image was captured
            }

    // Handles the event of a new image getting captured and visualizes it with the Image Visualizer
    private void OnCaptureRawImageComplete(MLCamera.CameraOutput capturedImage, MLCamera.ResultExtras resultExtras, MLCamera.Metadata metadataHandle)
    {
        Debug.Log("DisplayImage");
        imageVisualizer.OnCaptureDataReceived(resultExtras, capturedImage);
    }

zhhqu0305 · August 20, 2024, 6:58pm

Thanks! I think this solves my problem. A follow-up on this - if my project relies on Vuforia, the RGB camera feed and eye-tracking data of both eyes (positions and rotations), shall I temporarily avoid migrating to the OpenXR workflow? So far it worked fine (better than before I would say), but it would be nice if camera pose can be returned via the PixelSensor API - I still cannot get OpenXR and Vuforia working together.

Thanks!

zhhqu0305 · August 20, 2024, 9:05pm

A quick follow-up: does the meshing subcomponent take over the main camera? I'm building a project that requires meshing, and the following happened when adding the camera access code:

08-20 16:54:15.028  1047  3318  3775 W ServiceManager: Permission failure: com.magicleap.permission.CAMERA_CAPTURE_CVIP from uid=10107 pid=20837
08-20 16:54:15.029  1047  3318  3775 D ServiceManager: Permission check violation (permissive=0): com.magicleap.permission.CAMERA_CAPTURE_CVIP from uid=10107 pid=20837, Permission ARE enforcing.
08-20 16:54:15.029  1047  3318  3775 W ServiceManager: Permission failure: com.magicleap.permission.CAMERA_CAPTURE_CVIP from uid=10107 pid=20837
08-20 16:54:15.029  1047  3318  3775 D ServiceManager: Permission check violation (permissive=0): com.magicleap.permission.CAMERA_CAPTURE_CVIP from uid=10107 pid=20837, Permission ARE enforcing.
08-20 16:54:15.029  1047  3318  3775 W ServiceManager: Permission failure: com.magicleap.permission.CAMERA_CAPTURE_CVIP from uid=10107 pid=20837
08-20 16:54:15.029  1047  3318  3775 D ServiceManager: Permission check violation (permissive=0): com.magicleap.permission.CAMERA_CAPTURE_CVIP from uid=10107 pid=20837, Permission ARE enforcing.
08-20 16:54:15.041  1047  3318  3775 W ServiceManager: Permission failure: com.magicleap.permission.CAMERA_CAPTURE_CVIP from uid=10107 pid=20837
08-20 16:54:15.041  1047  3318  3775 D ServiceManager: Permission check violation (permissive=0): com.magicleap.permission.CAMERA_CAPTURE_CVIP from uid=10107 pid=20837, Permission ARE enforcing.

It only failed when I used the Main Camera to obtain the JPEG image (the CV camera only has YUV image), and using the CV camera for the RGBA video stream worked fine. I'm using the official Async code example, simply changing the configs from video to image only.

kbabilinski · August 20, 2024, 9:09pm

CAMERA_CAPTURE_CVIP is a signature permission and only available for systems apps/processes. Did you grant Spatial Mapping Permissions when you added the meshing component?

Topic		Replies	Views
Unable to get physical camera pose Unity Development camera-api , sensors	13	383	December 7, 2023
MLCVCamera.GetFramePose Failure Unity Development	14	26	October 7, 2024
GetFramePose is Not OK Unity Development openxr , camera-api	10	276	February 20, 2024
Camera capture timestamp delta Unity Development api , camera-api , camera	11	543	February 21, 2024
Unity sample for Camera2 API to capture image from CV/Main camera Unity Development unity , openxr , camera-api , question	2	98	July 30, 2024
Questions about Pixel/Camera API Unity Development unity , camera , eye-tracking	22	54	November 19, 2024

Camera API Compatibility with OpenXR, OS and Vuforia

Related topics