Meshing - how to tell by code if a surface is a wall or floor or furniture?

General
1

I have been putting together a proof of concept with ML2 recently and I have been using Unity and MRTK as I think it is a great tool for rapid prototyping.

I have been experiementing with meshing and was wondering how can we detect if my surface is a wall or a floor for example?

I know ML provide Plane detection from the example and api but this seem to be done with AR Foundation and an XR rig camera but since I am using MRTK, the camera and whole project setup is different and I was not able to use these feature in conjunction with MRTK. I am able to use the meshing example in conjunction with MRTK. I also noticed we can do this the MRTK way.
I was wondering:

  • Can we detect surface as a wall/floor type with the meshing feature?
  • If not can it be done with MRTK?
  • If not, are there any other way it can be done?
  • How would it work for furniture and any other type of surfaces?
1 Like
2

Hi @jc1. I'll talk to our engineers and get you an answer asap. Thanks for your post.

1 Like
3

@jc1 The Magic Leap 2 does not support plane finding via MRTK. Please use the AR Foundation APIs to detect the surface types of each plane.

You can add the XR Origin , AR Plane Manager , AR Session and AR Input Manager components to the MixedRealityPlayspace .

Make sure the Tracking Origin Mode is set to Device and the Y Offset is set to 0. and set a Plane Prefab inside the AR Plane Manager. Once this is complete you should see plane populating inside your scene.

4

One thing to note is that we noticed a bug in one of Unity's packages that may not report the correct planes correctly when using the AR Raycast Manager. We are working quickly to fix this, as a temporary workaround use the Plane Manager's Raycast directly. Here is an example of casting a ray from the controller and getting the hit type of collided plane. Note: the following sample was not analyzed for performance.

using System.Linq;
using Unity.Collections;
using UnityEngine;
using UnityEngine.InputSystem;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;

public class PlaneRaycastExample : MonoBehaviour
{
    private MagicLeapInputs mlInputs;
    private MagicLeapInputs.ControllerActions _controllerActions;
    private ARPlaneManager arPlaneManager;

    void Start()
    {
        arPlaneManager = FindObjectOfType<ARPlaneManager>();

        if (arPlaneManager == null)
        {
            Debug.LogError("Missing AR Plane Manager. Please make sure it is present in the scene hierarchy.");
            enabled = false;
            return;
        }

        mlInputs = new MagicLeapInputs();
        mlInputs.Enable();

        _controllerActions = new MagicLeapInputs.ControllerActions(mlInputs);
        _controllerActions.Bumper.performed += HandleOnBumper;
    }

    private void HandleOnBumper(InputAction.CallbackContext obj)
    {
        bool bumperDown = obj.ReadValueAsButton();
        Debug.Log("The Bumper is pressed down " + bumperDown);

        var raycastRay = new Ray(_controllerActions.Position.ReadValue<Vector3>(),
            _controllerActions.Rotation.ReadValue<Quaternion>() * Vector3.forward);

        var hits = arPlaneManager.Raycast(raycastRay, TrackableType.PlaneWithinPolygon, Allocator.Temp);

        if (hits.IsCreated)
        {
            using (hits)
            {
                var orderedHits = hits.OrderBy(_ => _.distance).ToList();
                for (int i = 0; i < 1; i++)
                {
                    Debug.Log("Raycast Hit " + orderedHits[i].hitType);
                }
            }
        }
    }
}