Remote Rendering - Unity - Hand Interaction

Remote Rendering
1

I am currently building an MRTK 2.8 application on Unity 2022.3.2f1. ML2 firmware version is 1.3.1.
I am using the camera profile OpenXRCameraProfile and the Input profile OpenXRInputSystemProfile.

In the OpenXR settings Hand Tracking and Hand Tracking Subsystem are enabled.

When I start remoting, the content is rendered as expected and head motion is correctly applied, however there seems to be no hand tracking. There are no hand visualizations and I am unable to interact with objects.

Any hints on what I'm missing is appreciated.

2

Hi Patrick,

Remote Rendering currently supports XR_EXT_hand_tracking but does not yet support XR_EXT_hand_interaction.

This is just a quick guess, but perhaps that is part of the issue?

We have yet to create much support documentation for Remote Rendering + Unity (With an initial focus on Unreal for Remote Rendering), but thanks to standard OpenXR things should work once you find the right levers within Unity. If you need further support, I can try to tag someone with further Unity expertise for you.

1 Like
3

Thanks for the hint, and I am looking forward to the documentation on how to use unity. However, it seems like I am stuck here.

I still get nothing from the hand tracking. If I set up controller support, I can see the controller pointer, but clicks don't do anything.

So, some additional hints from a unity guru would be appreciated.

4

Hi Patrick,

I'm afraid among our Remote Rendering experts, we don't have much in the way of Unity gurus. That said, here is the information I can provide, but please take this with a grain of salt as this comes from a few internal experiments.

  • We found that MRTK 3 worked better with Remote Rendering than MRTK 2 in our experiments.
  • In the OpenXR interaction profiles, enabling " Microsoft Hand Interaction Profile" might help, though this might actually still depend on some upcoming features (That should be available soon for Remote Rendering)

Sorry I can't be of better help here, but this is the current state of our RR + Unity knowledge. In the future, support for this will improve.

5

Also, updating your version of Remote Rendering might help with this, though even our upcoming Remote Rendering 1.7 version will not have the newer XR_EXT_hand_interaction extension. We are hoping to add support for it as soon as reasonable though.