Issues using 2.2.0 on 1.7.0, with MRTK3 and OpenXR

[localjoost]

I have just deployed an app using MRTK3 and noticed the following:

• Hand control works fine
• Controller does not work:
  • There is most of the time no ray coming from the controller
  • Although it is possible to activate the hover action of a button (as if an invisible ray is hitting it) actually activating the button is impossible. The odd thing is: the button acts like it is pressed, but the attached event is not triggered

This behavior can be observed in this app.

I have played with these settings but they seem to have no effect at all:

Interestingly, when I uncheck this checkbox, as it is done in the MRTK3 template project the controller works fine, but then the hands don't work.

How can I make both hands and controller work?

• Unity 2022.2.32f1
• OS 1.7.0
• Development on Windows.
• Using OpenXR (of course!)
• SDK 2.2.0
• com.magicleap.mrtk3: 1.0.0
• Latest MRKT3

[whebert]

Hey @localjoost,

Have you configured the rig for Magic Leap input, as described in the com.magicleap.mrtk3 package README?

If not already configured, I'd recommend enabling the runtime rig configuration option in settings, which just configures the default MRTK rig.

This should setup the input and add the Magic Leap Controller prefab to the rig, so you get both hands and controller.

Please let us know if that worked for you, or if you were already using one of the options and it still isn't working.

[localjoost]

Hi, good news and bad news. The good news is - yes, I had forgotten to check the bloomin' Runtime Config Enabled box, which is really stupid and I cannot imagine how I could have missed that. I did check the override box

The bad news is - it does not help a single iota. The device still does not ask for permission, not after reinstall, not after reinstall after a clean build. And consequently, it cannot find the floor.

Without this piece of code, it simply does not work and I have no idea why

#if MAGICLEAP
Permissions.RequestPermissions(new{
MLPermission.SpatialMapping} ,
OnPermissionGranted, OnPermissionDenied, OnPermissionDenied);
#endif

[whebert]

@localjoost , so does the Magic Leap Controller input now work at least, and you can use hands and controller at the same time?

As for the permissions, that is odd. Did you test removing/commenting out all manual permission request code in the project so as to not interfere with the auto permission requests? You could also possibly clone and build the OpenXR branch of the ML2 fork project and see if that works for you as a reference point. There is a spatial mapping example provided in the com.magicleap.mrtk3 package that would validate if spatial mapping request and feature is working.

[localjoost]

Hi,

I have more information on the controller. The MRTK3 has two kinds of buttons, canvas based and non-canvas based. The non-canvas buttons work with the controller and by pressing it with you finger. However, the Canvas buttons do only work by pressing with you finger, not the controller. The odd thing is, you see the graphic response and you hear the click sound but the OnClick event is not fired. You can easily repro this: just take your own GitHub - magicleap/MixedRealityToolkit-Unity: This repository holds the third generation of the Mixed Reality Toolkit for Unity. The latest version of the MRTK can be found here., build the HandInteractionExamples scene into an app. You will see almost all button properly work with the controller, but not those of the hand menu

If you TAP the top three buttons (with your finger), you will see the toggle move from left to right and back. If you point the controller to the button and press the trigger, the button moves backward, it makes the click sound but the toggle stays put. Interestingly, if I change the hand menu by moving the buttons further forward, press them once with my finger - after that I can actually operate them with the controller. For a short while. After that the controller seems to stop working all together. Long story short: I feel controller + MRTK3 + OpenXR is seriously unreliable. Whether or not that is a real problem is up to you. I think hand control is the way of the future anyway.

As to the automatic asking for permission, I removed the code to ask for permission again, removed the app and reinstalled it - does not ask for permission.
Upgraded to OS1.8.0, ML MRTK 1.1.0 and SDK 2.3.0, does not ask for permission. I do have to explicitly add code to do so, then it works. Well, I know see often a controller hanging at my hand.

I also built the SpatialMappingExample scene from GitHub - magicleap/MixedRealityToolkit-Unity: This repository holds the third generation of the Mixed Reality Toolkit for Unity. The latest version of the MRTK can be found here. into an app. That also has this set
. It does NOT ask for permissions and if I grab the ball and throw it down, it goes right through the floor to Australia, I presume. In other words: same problem. So using your own fork I was able to repro both problems.

[whebert]

Hey @localjoost,

Thanks for taking the time to clone and test the Magic Leap MRTKDevTemplate fork project to see if it helps with the issues you are experiencing.

• Regarding the Controller issue on Canvas elements:

It could be that the hand menu toggle button "Hand Rays", was toggled off when you were trying to click things with the Controller ray? Unfortunately, the logic for the hand menu "Hand Rays" toggle button disables all of the MRTKRayInteractors in the scene, which would include our ML Controller's ray interactor. It makes no distinction of what type of controller (hand or otherwise) the MRTKRayInteractor belongs to (code below).

Perhaps we can work to get that toggle logic to be more selective upstream in the MRTK repo to only affect hand controllers. But please try Canvas elements with the ML Controller while the hand menu "Hand Rays" toggle is enabled, it should work as usual.

• Regarding the Permissions and the SpatialMappingExample scene.

Setting aside the permission issue briefly, that built-in SpatialMappingExample scene that comes with the MRTK project should work (assuming permissions work), but only if you are using the OpenXR provider. Also, you have to set the OpenXR render mode to mult-pass in order for the wireframe shader to work on the ML2.

If you'll be testing with OpenXR and the ML fork of the MRTKDevTemplate project, be sure to checkout the mrtk3_MagicLeap2_OpenXR branch of the fork, which should have all the OpenXR configuration setup correctly.

Make sure the Magic Leap 2 Meshing Subsystem OpenXR feature is selected, among other things. The below config is how the above branch is configured for the features.

The spatial mapping test scene I was referring to in an earlier post, is one we provide as a sample within the com.magicleap.mrtk3 package itself. Import the "Spatial Awareness Examples" sample from the package, and run that scene. That meshing scene works with either provider and has a wireframe renderer that doesn't require the multi-pass selection.

Of course all that assumes permissions work, which you have working with manual code at the moment. The permissions worked for me just now in a fresh test of the fork, in either XR Provider branch. But I am using an earlier Unity version. I'll install and test the specific version of Unity you are using to see if it makes any difference.

[localjoost]

Hi @whebert ,

Yeah, I have double checked all the settings. In addition, I have imported the sample SceneReconstruction into the MRTK project and this was my result

Permission pending, and never got asked. Even though I have set everything right IMHO

For the record, I think for Spatial Mapping to work at all no also need to tick the Magic Leap 2 Experimental checkbox, as meshing is now suddenly 'experimental', which is was not in ML MRTK 1.0.0. This is quite a bit confusing as it disabled meshing in my own app even if I explicitly asked for permission

So it might all come down to a Unity version. We all know Unity is like move fast, break things, let others take care of the fixing

I also, all modesty, wish to state the SceneReconstruction is not quite a good example, as it already has the MRTK XR Rig - MagicLeap in it, in stead of the regular MRTK XR Rig, which should make the "Runtime Config Enabled" superfluous. As this is one of the most powerful tools to make a cross platform app, think this might put people on the wrong footing.

Kind regards,

Joost

[localjoost]

@whebert

Regarding the controller: I have no idea what is going on here. Hand ray air tap works, pressing works, controller does not. I can't upload an MP4 so here's a link: controller.mp4

I also have tried to move buttons way before the 'screen' but that did not help.
I also have messed with this in my app Walk The World before I released it, but I could not see any effect whatsoever from changing this option.

I think my earlier experience with having the button sometimes respond to the controller in the MRTK fork was actually the device registering an air tap while I was pressing the controller

[svc_discoursezendesk]

@whebert

Regarding the controller: I have no idea what is going on here. Hand ray air tap works, pressing works, controller does not. I can't upload an MP4 so here's a link: controller.mp4

I also have tried to move buttons way before the 'screen' but that did not help.
I also have messed with this in my app Walk The World before I released it, but I could not see any effect whatsoever from changing this option.

image825×268 24.6 KB

[Discourse post]

[localjoost]

@whebert

Taking the chance you get completely fed up with me, I could not resist doing more research on the controller. If I replace the MRTK XR Rig by the MRTK XR Rig - MagicLeap - OpenXR in my scene the controller works as expected - I can operate buttons it, be it canvas or non canvas. I also get a nice ghost-transparent-gray hand visualization, and I get to see the correct Magic Leap 2 controller model, in stead of something that suspiciously look like a Quest Pro 3 controller. My guess: the Runtime Config Enabled thingy does not work correctly or completely with Open XR.

Oh, and in case you wonder - I still had to manually ask for spatial map permissions.

For the record: now using these Magic Leap package versions

[whebert]

@localjoost,

Yes, if the MRTK XR Rig - MagicLeap -OpenXR rig variant is working, but the default MRTK XR Rig is not when the Runtime Rig Config option is selected, that sounds like something odd is going on with that runtime rig config option and it isn't working as expected - you seeing the default controller models and not the hand mesh also indicates this is the case. This works fine when I test it with our fork, in either XR provider branch, so it should work for you - no idea why not yet. (The reason we use a ML rig variant in the samples is that we can't guarantee the developer will have enabled the runtime rig configuration option in settings, so in order to make sure the samples work, we use our rig variant)

We appreciate you taking the time to test things in attempt to get to the bottom of the issues you're experiencing. Would it be possible for you to push up a project to github, even just a minimal project which reproduces the issue? That way we could pull it down and take a look at the configuration, try to help determine the issue and exactly what is going on.

That may be the best way for us to help figure out what may be happening, as in our own fork with our own testing, things seem to work as expected. I did even try Unity version 2022.3.32f1, and everything still seems fine for me, even permissions.

[whebert]

@localjoost,

I may have just spotted the issue. I took a closer look at your screenshots above and noticed that they are showing settings for the "Magic Leap" XR Provider, not the OpenXR Provider. Since we support both XR providers, which are similar but can have slightly different options - and which one may be currently active depends on if running in Editor Play mode or being built to a platform, we provide a dropdown at the top of our ML MRTK3 settings to select which provider the settings are for.

Part of your screen shot above (note the XR Provider: MagicLeap):

Choose the OpenXR provider at the very top of the settings to make sure you are setting up the options for the OpenXR Provider like so:

This step is described in our README when configuring the rig options for the OpenXR provider:

Hopefully that is the issue you are experiencing, at least with the rig configuration. This would not affect the permissions as that section is shared by either provider. So something odd is still going on there. It would still be helpful to share a github project if possible to help investigate.

[localjoost]

Hi @whebert ,

Unfortunately, that screenshot was from the MRTK3 fork settings. In my own project, I don't even have the opportunity to choose anything but OpenXR:

[whebert]

@localjoost,

That sounds right if your project doesn't have the ML XR Provider package (com.unity.xr.magicleap). And if you are solely using OpenXR, there is no need to have that legacy package, so only the OpenXR settings would show up.

But if you are still experiencing the rig configuration issues under OpenXR, I'd say the best option would be to hopefully share a github repo that we could pull down and test to help investigate.

[localjoost]

Here you go @whebert :

LocalJoost/MLRepro: Repro for not-working auto permission (github.com)

It asks you to look at the floor. If you do so and the floor is found, it shows a red square if you are looking at the floor but too far away, if you look close enough it will turn green.

If you deploy the app with the check box "Ask For Permission" off, it will not work. I suggest you try that first. Then check the "Ask For Permission" checkbox, deploy again, it then asks for permission, and it works. The spatial mapping stuff is handled by an ARMeshManager on the Camera Offset.

For the rest this is a bog standard MRTK XR Rig

This is basically the code I use in Walk the World, but I pried it rather brutally loose from the app logic, so it may look a bit odd - my coding standard usually is a bit higher, but I did not want to obscure this behind a service oriented UniRx based architecture.I hope it is clear enough to demonstrates the issue and for you to find out what I am doing wrong.

[whebert]

@localjoost,

Thanks for the github repo to test this with. So I cloned your repo fresh and built the project without any changes, and it appears to work for me. Both permissions (the dialog comes up) and the rig configuration seem to work fine (hand meshes and ML Controller). I captured some screen shots (since it isn't easy to share video here) below, showing the permission dialog come up, and showing hand mesh and controller model, and switching hands holding controller. So I take it this is not what you see when you build it on run on your device?

[localjoost]

You are right, I don't see that. Also, did the square get green when you looked right before your eyes?

Anyway, humor me: this is the Apk I built. De-install your compiled project, and install mine. It is without the explicit ask-for-permission.

[whebert]

@localjoost,

OK, I uninstalled my built APK, downloaded yours and installed that. I continue to see the exact same thing. The permission dialog comes up, and then the red square and message to look at floor, with hand meshes and controller model all working. And as in the images above, the room mesh is obscuring the square and message depending on where I look.

But the square stays red the whole time, never turns green for me - what is supposed to make it turn green?

[localjoost]

@whebert

You might have noticed the the square following your head ray. If you stand up and look like basically just before your feet it should become green. But anyway - the fact that you even see the square shows that it works.

I am kind of out of ideas here. I suppose it's just my device, then? What on Earth can be wrong with it that it does not honor the automatic permission thing?

[whebert]

@kbabilinski Please see above.

Do you have any ideas why @localjoost and I would be seeing different results with the same APK? Is there perhaps a Settings change that could affect any of this?

Also, @localjoost, you aren't seeing the hand mesh and controller model as well, right?