I'm trying to get the XR_ML_world_mesh_detection extension working in my application. I followed the sample code 1 provided in the extension with very few modification, and it seems like the mesh detection step is stalling the whole application.
As you can see in the log beneath, there seems to be an issue with the timestamp in xrRequestWorldMeshStateCompleteML causing the stall.
log:491:11-06 18:50:38.264 13284 13306 V REDACTED: [/REDACTED/mesh_detector.c:118] query mesh info
log:492:11-06 18:50:38.264 13284 13306 V REDACTED: [/REDACTED/mesh_detector.c:126] xrRequestWorldMeshStateCompleteML
log:493:11-06 18:50:38.264 13284 13306 V REDACTED: [/REDACTED/mesh_detector.c:133] xrRequestWorldMeshStateCompleteML done at 0
log:494:11-06 18:50:38.264 13284 13306 I REDACTED: [/REDACTED/mesh_detector.c:135] completion futureResult: XR_SUCCESS
log:496:11-06 18:50:38.264 13284 13306 V REDACTED: [/REDACTED/mesh_detector.c:142] mesh_count 0 meshBlockStateCountOutput 429
log:497:11-06 18:50:38.264 13284 13306 V REDACTED: [/REDACTED/mesh_detector.c:144] realloc mesh_states & mesh_blocks
log:498:11-06 18:50:38.264 13284 13306 V REDACTED: [/REDACTED/mesh_detector.c:163] xrRequestWorldMeshStateCompleteML
log:556:11-06 18:50:42.173 13284 13306 E com.REDACTED: leapcore/frameworks/perception/data_sources/include/pad/xpad_data_source.h(105) GetClosestTimestampedData():
log:557:11-06 18:50:42.173 13284 13306 E com.REDACTED: ERR: Data Not Found for timestamp: 10416275us, now time: 10421734908us
log:558:11-06 18:50:42.174 13284 13306 V REDACTED: [/REDACTED/mesh_detector.c:169] xrRequestWorldMeshStateCompleteML done at 10416275399
The timestamp printed by GetClosestTimestampedData seems a little off compared to the now time, could this be the issue?
The native meshing api doesn't seems to be affected by the same issue, I'm able to get a world mesh without stalling everything. I'm updating the mesh_detector between xrWaitFrame and xrBeginFrame, only if XrFrameState::shouldRender is true and the session is visible or focused.
Sorry for the delay. We'll see if we can reproduce the performance issue you describe here. If you have a relatively self-contained code sample that you can share, we'd be happy to take a look through your code and try running it. I haven't observed problems after some quick testing with applications that use the OpenXR world mesh apis, but that doesn't mean there isn't an issue.
In particular, it might be interesting to take a look at your frame timing loop if you're making calls to the world mesh apis on the same thread. The mesh apis are meant to be async, but there might be something there.
By the way, just for sanity, by "The native meshing api doesn't seems to be affected by the same issue" are you referring to the MLMeshing C apis? Were you using the MLMeshing apis in the same OpenXR application, or were you using the ML C apis exclusively? If it's easy enough to do, it might be interesting to compare your implementation using the OpenXR meshing extension to your previous implementation. We can send you a dm or setup a private channel if you'd rather not share code publicly.
Here are both header and source from my application, beside a few logging headers and the xr_linear.h header from the OpenXR-SDK, it should be fairly self contained.
As for frame loop, I update the mesh detector between xrWaitFrame and xrBeginFrame only if XrFrameState::shouldRender is XR_TRUE and the session state is either XR_SESSION_STATE_VISIBLE or XR_SESSION_STATE_FOCUSED.
I though about moving the mesh_detector in its own thread but haven't done it yet.
Yes I'm refering to the MLMeshing C api, which I also tried in parallel (exclusively), you can see the issues I found with it here. I can also share the code if you'd like.
We have found where your application has stalled and we are working with our team to find the root cause of this issue. I will report back as soon as we learn more.
Thanks for your patience with this issue! We tried profiling your code sample and the world mesh api calls were running far longer than would be expected-
There is currently an issue where the api calls will hang if you attempt to query for meshes relative to VIEW space. We're reported the issue to our internal teams. In the mean time, would you be able to query for meshes relative to XR_REFERENCE_SPACE_TYPE_LOCAL or XR_REFERENCE_SPACE_TYPE_UNBOUNDED_MSFT and see if you observe similar performance problems?
At first I though that using XR_REFERENCE_SPACE_TYPE_LOCAL would fix the issue on my side. I reworked the mesh_detector file to process the vertex data, but I noticed weird a behaviour.
First, a lot of block states have the XR_WORLD_MESH_BLOCK_STATUS_DELETED_ML for states that I haven't seen before. Then the same blocks have XR_WORLD_MESH_BLOCK_RESULT_FAILED_ML as blockResult. The rare block states that have a different status also all have XR_WORLD_MESH_BLOCK_RESULT_FAILED_ML as blockResult.
No blocks are rendered by my application.
Here are the new and updated mesh_detector files mesh_detector.zip (3.8 KB)
And an execution log of the application log.txt (1.9 MB)
Sorry for the slow follow up here. At a glance, the results you presented do seem unusual. World mesh OpenXR apis are flexed though our Unreal sdk, so we should be able to determine what's going on. (the sdk is distributed as source btw, you can download the 'Unreal Plugin' package from ML hub if you want to take a look at the world mesh plugin implementation) We'll take a closer look and do some testing with the code sample that you sent over. In the mean time, if you wanted to send over complete source for a runnable application, then that might help accelerate things a bit.
Thanks for your quick answer @aheaney, is there a place I can DM or email you? I can send an archive of the source with both the ML and the XR meshing api and build directives.
