How to use the hand interaction like the system menu?

Hello there,
I'm developing my own native application on Magic leap 2 that should have far interaction with hands, but I've an issue when I try to set the pointer pose to interact with my scene objects. It doesn't look the same as the pointer pose when I interact with the system menu. In the following gif images, you can find the different behaviours of the pointers between my application and the system menu.

ezgif-3-eb7ca1c408
ezgif-3-906254bde5

I'm using the hand_interaction array that is documented here in your Native API documentations.

should I do any kind of transformations to get the correct pointer pose values ? or should I use something else to get it ?

could you guide me to fix this issue ?

ML2 OS version: 1.3.0-dev2
MLSDK version: 1.3.0-dev2

Thank you in advance

Hi @m.soliman,

Welcome to the Magic Leap 2 developer forums. We are so grateful to have you here engaging with us.

I would recommend using the hand_transform CFUID instead of the hand_interaction. Try using MLGestureClassificationPostureType to get the pinch gesture for selecting.

Let me know if this is helpful.

Best,

El

I have submitted your feedback regarding access to the hand rays that we use in our OS. In the meantime I recommend referencing the logic for the MRTK pointer

This pointer uses a combination of hand and head tracking to create user friendly input. Let us know if you have any trouble converting any of the logic to C++.

Unfortunately, that didn't solve the issue. It just changed the source joint of the ray almost to the center of the hand but still the ray doesn't look like the ray in the system menu.

Any more suggestions please ?

Maybe my question was not clear enough but actually I'm trying to get the correct value of the pointer pose which should fix my issue with the hand ray according to my app logic (I'm not drawing the hand ray by myself). I gave up to get the correct value of the pointer pose from the C-API and now I switched to use OpenXR which I would prefer more to use. So now I'm trying to use 12.29. XR_EXT_hand_interaction to get the pointer pose and the pinch actions.
I can already see Magic Leap in the Contributors List, but it is not mentioned in the list of supported extensions in the Magic Leap Docs, so I am not sure if the runtime currently supports this extension.
When I try to use that extension, I get a warning that this extension is not found and also cannot find any release notes about your OpenXR runtime release notes on the developer-docs.

could you please provide more information about OpenXR runtime on Magic Leap 2 ?
or any more suggestions to get fix the hand interaction issue ?

Thanks in advance

I am working on getting more information regarding the support of the Hand Interaction profile. While I track it down, you can see a list of the supported extensions see our OpenXR documentation

Thank you for the feedback regarding the OpenXR runtime release notes. These will be included on our developer portal in future releases. In the meantime you can see the release notes in the ML Hub under the OpenXR Samples:

Hi @m.soliman I have confirmed that the pointer pose will be supported in the upcoming release which will be available soon. Keep an eye out on the Release newsletter and release notes for more information.

Hello again,

With the last release 1.4.0-dev1, I could grab the pointer pose for both hands using the hand interaction extension XR_EXT_hand_interaction. And, it worked completely fine with the right hand. However I had an issue with the left hand where the pointer was oriented downwards as you can see in the following image:

As a feedback I managed to fix the left pointer pose by doing some rotations around X and Y axes as the following:

I changed the orientation by rotating 45 degrees around X-axis clock-wise and 15 degrees around Y-axis counter-clock-wise and that gave me a smoother pointer on the left hand as you can see in the following image:

I would like to ask if there is any reason for not having the left pointer pointing forwards like the right pointer.

Regards

Thank you @m.soliman,

This appears to be a bug, thank you for brining it to our attention.

Hi @kbabilinski,

I've upgraded the OS to v1.4.0-dev2 and that fixed the rotation issue for the left hand.

Thank you

1 Like