XR_ML_world_mesh_detection timing issue

Hello there,

I'm trying to get the XR_ML_world_mesh_detection extension working in my application. I followed the sample code 1 provided in the extension with very few modification, and it seems like the mesh detection step is stalling the whole application.

As you can see in the log beneath, there seems to be an issue with the timestamp in xrRequestWorldMeshStateCompleteML causing the stall.

log:491:11-06 18:50:38.264 13284 13306 V REDACTED: [/REDACTED/mesh_detector.c:118] query mesh info
log:492:11-06 18:50:38.264 13284 13306 V REDACTED: [/REDACTED/mesh_detector.c:126] xrRequestWorldMeshStateCompleteML
log:493:11-06 18:50:38.264 13284 13306 V REDACTED: [/REDACTED/mesh_detector.c:133] xrRequestWorldMeshStateCompleteML done at 0
log:494:11-06 18:50:38.264 13284 13306 I REDACTED: [/REDACTED/mesh_detector.c:135] completion futureResult: XR_SUCCESS
log:496:11-06 18:50:38.264 13284 13306 V REDACTED: [/REDACTED/mesh_detector.c:142] mesh_count 0 meshBlockStateCountOutput 429
log:497:11-06 18:50:38.264 13284 13306 V REDACTED: [/REDACTED/mesh_detector.c:144] realloc mesh_states & mesh_blocks
log:498:11-06 18:50:38.264 13284 13306 V REDACTED: [/REDACTED/mesh_detector.c:163] xrRequestWorldMeshStateCompleteML
log:556:11-06 18:50:42.173 13284 13306 E com.REDACTED: leapcore/frameworks/perception/data_sources/include/pad/xpad_data_source.h(105) GetClosestTimestampedData():
log:557:11-06 18:50:42.173 13284 13306 E com.REDACTED: ERR: Data Not Found for timestamp: 10416275us, now time: 10421734908us
log:558:11-06 18:50:42.174 13284 13306 V REDACTED: [/REDACTED/mesh_detector.c:169] xrRequestWorldMeshStateCompleteML done at 10416275399

The timestamp printed by GetClosestTimestampedData seems a little off compared to the now time, could this be the issue?

The native meshing api doesn't seems to be affected by the same issue, I'm able to get a world mesh without stalling everything. I'm updating the mesh_detector between xrWaitFrame and xrBeginFrame, only if XrFrameState::shouldRender is true and the session is visible or focused.

I can provide more code if requested.

Thanks.

ML2 OS version: 1.8.0
MLSDK version: 1.10

Hi @simon.zeni,

Welcome to the Magic Leap 2 Forums. We are so grateful for your engagement here.

I am going to try to reproduce this issue using this example and I will report back as soon as I learn more.

Hello,

Sorry for the delay. We'll see if we can reproduce the performance issue you describe here. If you have a relatively self-contained code sample that you can share, we'd be happy to take a look through your code and try running it. I haven't observed problems after some quick testing with applications that use the OpenXR world mesh apis, but that doesn't mean there isn't an issue.

In particular, it might be interesting to take a look at your frame timing loop if you're making calls to the world mesh apis on the same thread. The mesh apis are meant to be async, but there might be something there.

By the way, just for sanity, by "The native meshing api doesn't seems to be affected by the same issue" are you referring to the MLMeshing C apis? Were you using the MLMeshing apis in the same OpenXR application, or were you using the ML C apis exclusively? If it's easy enough to do, it might be interesting to compare your implementation using the OpenXR meshing extension to your previous implementation. We can send you a dm or setup a private channel if you'd rather not share code publicly.

Best,
Adam

Hi @aheany, don't worry for the delay.

Here are both header and source from my application, beside a few logging headers and the xr_linear.h header from the OpenXR-SDK, it should be fairly self contained.

mesh_detector.zip (2.9 KB)

As for frame loop, I update the mesh detector between xrWaitFrame and xrBeginFrame only if XrFrameState::shouldRender is XR_TRUE and the session state is either XR_SESSION_STATE_VISIBLE or XR_SESSION_STATE_FOCUSED.

I though about moving the mesh_detector in its own thread but haven't done it yet.

Yes I'm refering to the MLMeshing C api, which I also tried in parallel (exclusively), you can see the issues I found with it here. I can also share the code if you'd like.

Just posting here to let you know that we are still working through your issue. We will report back as soon as we learn more.

Hi @simon.zeni,

We have found where your application has stalled and we are working with our team to find the root cause of this issue. I will report back as soon as we learn more.

Thanks for keeping me updated @etucker I hope you'll get to the bottom of this soon!

Hi @simon.zeni ,

Thanks for your patience with this issue! We tried profiling your code sample and the world mesh api calls were running far longer than would be expected-

There is currently an issue where the api calls will hang if you attempt to query for meshes relative to VIEW space. We're reported the issue to our internal teams. In the mean time, would you be able to query for meshes relative to XR_REFERENCE_SPACE_TYPE_LOCAL or XR_REFERENCE_SPACE_TYPE_UNBOUNDED_MSFT and see if you observe similar performance problems?

Thanks!
Adam

1 Like

Hi @aheaney

Using XR_REFERENCE_SPACE_TYPE_LOCAL as meshSpace seems to work fine. I get a different number of mesh blocks (but not 0).

I'll wire up the rendering for the XR mesh detection and see if the meshes make sense. Thanks.

Hi again @aheaney

At first I though that using XR_REFERENCE_SPACE_TYPE_LOCAL would fix the issue on my side. I reworked the mesh_detector file to process the vertex data, but I noticed weird a behaviour.

First, a lot of block states have the XR_WORLD_MESH_BLOCK_STATUS_DELETED_ML for states that I haven't seen before. Then the same blocks have XR_WORLD_MESH_BLOCK_RESULT_FAILED_ML as blockResult. The rare block states that have a different status also all have XR_WORLD_MESH_BLOCK_RESULT_FAILED_ML as blockResult.

No blocks are rendered by my application.

Here are the new and updated mesh_detector files mesh_detector.zip (3.8 KB)
And an execution log of the application log.txt (1.9 MB)

Thanks.