How to use the hand interaction like the system menu?

m.soliman · August 8, 2023, 2:46pm

Hello there,
I'm developing my own native application on Magic leap 2 that should have far interaction with hands, but I've an issue when I try to set the pointer pose to interact with my scene objects. It doesn't look the same as the pointer pose when I interact with the system menu. In the following gif images, you can find the different behaviours of the pointers between my application and the system menu.

ezgif-3-eb7ca1c408
ezgif-3-906254bde5

I'm using the hand_interaction array that is documented here in your Native API documentations.

should I do any kind of transformations to get the correct pointer pose values ? or should I use something else to get it ?

could you guide me to fix this issue ?

ML2 OS version: 1.3.0-dev2
MLSDK version: 1.3.0-dev2

Thank you in advance

etucker · August 8, 2023, 5:37pm

Hi @m.soliman,

Welcome to the Magic Leap 2 developer forums. We are so grateful to have you here engaging with us.

I would recommend using the hand_transform CFUID instead of the hand_interaction. Try using MLGestureClassificationPostureType to get the pinch gesture for selecting.

Let me know if this is helpful.

Best,

El

kbabilinski · August 9, 2023, 6:46pm

I have submitted your feedback regarding access to the hand rays that we use in our OS. In the meantime I recommend referencing the logic for the MRTK pointer

github.com

microsoft/MixedRealityToolkit-Unity/blob/81aa4e9799c2b28194c9e8aea9a2701226087b2e/Assets/MRTK/Core/Providers/Hands/HandRay/HandRay.cs#L9


      
          // Copyright (c) Microsoft Corporation.
          // Licensed under the MIT License.
          
          using Microsoft.MixedReality.Toolkit.Utilities;
          using UnityEngine;
          
          namespace Microsoft.MixedReality.Toolkit.Input
          {
              public class HandRay : IHandRay
              {
                  /// <inheritdoc />
                  public Ray Ray
                  {
                      get
                      {
                          ray.origin = stabilizedRay.StabilizedPosition;
                          ray.direction = stabilizedRay.StabilizedDirection;
                          return ray;
                      }

This pointer uses a combination of hand and head tracking to create user friendly input. Let us know if you have any trouble converting any of the logic to C++.

m.soliman · August 17, 2023, 8:21am

Unfortunately, that didn't solve the issue. It just changed the source joint of the ray almost to the center of the hand but still the ray doesn't look like the ray in the system menu.

Any more suggestions please ?

m.soliman · August 17, 2023, 8:35am

Maybe my question was not clear enough but actually I'm trying to get the correct value of the pointer pose which should fix my issue with the hand ray according to my app logic (I'm not drawing the hand ray by myself). I gave up to get the correct value of the pointer pose from the C-API and now I switched to use OpenXR which I would prefer more to use. So now I'm trying to use 12.29. XR_EXT_hand_interaction to get the pointer pose and the pinch actions.
I can already see Magic Leap in the Contributors List, but it is not mentioned in the list of supported extensions in the Magic Leap Docs, so I am not sure if the runtime currently supports this extension.
When I try to use that extension, I get a warning that this extension is not found and also cannot find any release notes about your OpenXR runtime release notes on the developer-docs.

could you please provide more information about OpenXR runtime on Magic Leap 2 ?
or any more suggestions to get fix the hand interaction issue ?

Thanks in advance

kbabilinski · August 22, 2023, 1:00am

I am working on getting more information regarding the support of the Hand Interaction profile. While I track it down, you can see a list of the supported extensions see our OpenXR documentation

Thank you for the feedback regarding the OpenXR runtime release notes. These will be included on our developer portal in future releases. In the meantime you can see the release notes in the ML Hub under the OpenXR Samples:

kbabilinski · August 22, 2023, 10:49pm

Hi @m.soliman I have confirmed that the pointer pose will be supported in the upcoming release which will be available soon. Keep an eye out on the Release newsletter and release notes for more information.

m.soliman · September 4, 2023, 9:56am

Hello again,

With the last release 1.4.0-dev1, I could grab the pointer pose for both hands using the hand interaction extension XR_EXT_hand_interaction. And, it worked completely fine with the right hand. However I had an issue with the left hand where the pointer was oriented downwards as you can see in the following image:

As a feedback I managed to fix the left pointer pose by doing some rotations around X and Y axes as the following:

I changed the orientation by rotating 45 degrees around X-axis clock-wise and 15 degrees around Y-axis counter-clock-wise and that gave me a smoother pointer on the left hand as you can see in the following image:

I would like to ask if there is any reason for not having the left pointer pointing forwards like the right pointer.

Regards

kbabilinski · September 5, 2023, 7:50pm

Thank you @m.soliman,

This appears to be a bug, thank you for brining it to our attention.

m.soliman · October 6, 2023, 1:21pm

Hi @kbabilinski,

I've upgraded the OS to v1.4.0-dev2 and that fixed the rotation issue for the left hand.

Thank you

Topic		Replies	Views
Binding hands interaction actions with the controller actions causing issues OpenXR development	4	324	October 6, 2023
Hand Tracking -MRTK3 - hands wrong direction and seem to be missing ray casts Unity Development	7	379	September 22, 2023
Problems trying to access hand pinch and aim transforms using input actions Unity Development unity , hand-tracking , input , xri	3	131	June 4, 2024
Hands are rotated incorrectly with MRTK + OpenXR MRTK Development unity	9	47	November 1, 2024
Hand tracking palm rotation not working as expected Unity Development	6	455	October 31, 2022

How to use the hand interaction like the system menu?

Related topics