Detecting Spherical Passive Reflective IR Markers

Unity Editor version: 6000.0.23f1
ML2 OS version: 1.11.0
Unity SDK version: 2.6.0-pre.R15
Host OS: (Windows/MacOS) Windows 11

Hi everyone,

I’m looking to track spherical passive infrared reflective markers, similar to the implementation described in this project for the HoloLens 2: DINO-Unity on GitHub.

Specifically, I’m wondering if it’s possible to use the Depth Sensor to detect the IR reflective material on these markers.

Can the Depth Sensor be leveraged for this purpose? If not, are there alternative approaches you’d recommend for tracking such markers?

I tried the Unity Example projects for WorldCamera and DepthCamera with different configurations without any luck.

Any suggestions, guidance, or resources would be greatly appreciated.

Thank you in advance!

Hi @tkCADS I'm not directly familiar with the project that you shared but you should be able to obtain the raw image from the ToF sensor and use it to detect the reflective IR markers.

Note, we have had feedback that the PixelSensor API does not provide the flexibility that some developers need since you are NOT able to obtain the Raw Image and the Filtered image data at the same time.

Here is some additional information about the Depth Sensor Configurations:

Capability Description
Range Modes Long Range (1m - 5m): Operates at 1 or 5 fps
Short Range (0.20m - 1.00m): Operates at 5 or 30 fps (60 fps is supported but can cause overheating when running for extended periods)
Resolution 544 x 480 pixels
Data Options Developers can query Depth Data or Raw Depth Data
Exposure Adjustment Developers can adjust the Exposure Time
Depth Range Developers can clamp the depth range for either mode

Depending on the selected output the following Metadata is available for each frame:

Configuration Metadata
Depth 32 Output - Depth Confidence Buffer
- Depth Flag Buffer
- Exposure Time
- Pinhole Camera Model
Raw Depth Output - Depth Frame Illumination
- Exposure Time
- Pinhole Camera Model

The Sensor's Position and Rotation can be queried regardless of configuration

Thanks for the quick and detailed response!

Just to clarify, I am looking for a way to achieve an infrared response that allows me to clearly distinguish between the infrared reflective materials and non-reflective materials.

To give you a better idea of what I’m aiming for, I’ve attached an image that illustrates my desired outcome:

In your opinion, which configuration and metadata response would be best suited for this purpose?

Thanks again!

I think that would be Depth Frame Illumination, but I'm not familiar with this subject.

If you have a Magic Leap 2 device you can see the latest Unity Example Project , it comes with a Pixel Sensor Example scene. Alternatively we have a Pixel Sensor Section on the Developer Documentation.

I was able to work out the IR response using DepthRaw—thanks for the guidance! However, as you previously mentioned, it’s not possible to access both DepthRaw and Depth32 simultaneously. I have a few questions about potential workarounds:

  • Is there a way to extract depth data from DepthRaw?
  • Can the Depth32 buffer provide an IR response, perhaps by leveraging the DepthFrameIllumination option somehow?
  • I assume switching between the two buffers every frame would not work due to latency—can you confirm?
  • Are there any alternative methods to obtain both Depth and IR response data at the same time?

I think it would be impossible for you to obtain depth using the DepthRaw image because:

Raw Depth Image : This is the raw depth camera sensor data captured with the depth camera illumination and corresponds to the amount of total light incident on the sensor.

vs

Depth Image: stores the Depth Data from the Depth Camera. Depth is represented in meters

While not on our roadmap, I will put in a feature request to the Voice of Customer team based on your requirements.