Obtaining Depth and Ambient Depth Images in Unity Example

Unity Development
, ,
1

I am having difficulty figuring out how to access the ambient raw depth image and non-ambient raw depth image from the ML2 in Unity C#.

I am testing out of the MagicLeapUnityExamples project, so solutions within the DepthCameraExample.cs or DepthStreamVisualizer.cs example scripts would be fantastic.

I know from this page in the docs that the variables below are the variables I want to access. However, I am not seeing the means to access their struct for MLDepthCameraFrame based on the Unity example or the docs, only the PixelSensorFrame, unfortunately.

Type / Name Description
MLDepthCameraAmbientRawDepthImage * ambient_raw_depth_image Ambient raw depth image. The illuminator in the sensor is modulated with a system determined frequency. This is the raw sensor data captured when the illuminator is off.
MLDepthCameraRawDepthImage * raw_depth_image Raw depth image. The illuminator in the sensor is modulated with a system determined frequency. This is the raw sensor data captured when the illuminator is on.

Any help is much appreciated!

Unity Editor version: 2022.3
ML2 OS version:
Unity SDK version: 2.5.0
Host OS: Windows 10

2

Edit: I forgot that we had added DepthCameraExample.cs and associated scripts from this doc into the same PixelSensor folder as the PixelSensorExample.cs script when starting this project.
I also forgot to add these example’s link in the main post (and can’t edit it apparently), so I’m doing it now!

3

Unfortunately, You can NOT access both the raw and depth stream from the depth sensor using the Pixel Sensor API

4

Ok, then what is the normal procedure for accessing MLDepthCameraFrame, so I may access ambient_raw_depth_image and raw_depth_image?

Would I access it from outside the PixelSensorFrame struct?

5

The MLDepthCamera API that you linked is part of the MLSDK which has been deprecated. The Pixel Sensor API is the OpenXR replacement however it does not support accessing both Depth and Ambient images.

6

Ah, I see…so my options are to use the MLSDK API to get both, or only the combined depth image with Pixel Sensor API in OpenXR - is that correct?

Are there any plans to expand the OpenXR package to be able to access both of the raw images?

7

Unfortunately, this functionality is not on our roadmap but I did tag your ticket as a feature request to track the functionality developers are looking for.

1 Like
8

Ok, thanks!

Final follow-up question: can the MLSDK API be used alongside the OpenXR API [safely]?

9

Some of the APIs, yes : Additional Features | MagicLeap Developer Documentation

You can use the MLSDK Depth API however it is experimental and has known issues that will not be resolved due to the SDK being depricated.

1 Like
10

Appreciated!

Follow-up question: How do we actually get raw/brightness depth frames from the MLDepthCamera using MLSDK ?

We cannot find a depth example for MLSDK, and we keep getting a timeout failure from GetLatestDepthData and are not seeing what is missing.

We have successfully:

  • Gained permission for the Depth Camera with Permissions.RequestPermission(Permissions.DepthCamera,...etc)

  • Connected with MLDepthCamera.Connect()

  • Enabled short range with
    var settings = MLDepthCamera.CurrentSettings;
    settings.Streams = MLDepthCamera.Stream.ShortRange; MLDepthCamera.SetSettings(settings);
    MLDepthCamera.UpdateSettings(settings);

We are not able to:

  • Set up MLDepthCamera.Settings.StreamConfig, as it is null, and apparently cannot be set, according to the documentation?

What else do we need to do? Thanks.

11

Unfortunately, the MLSDK is deprecated and was experimental at the time. Because of this, there are outstanding bugs and the API is not recommended for production use.

The old version of the Magic Leap examples (1.12.0) may include the example for the API but I’m not sure.