Unity Editor version: 6000.0.23f1 ML2 OS version: 1.11.0 Unity SDK version: 2.6.0-pre.R15 Host OS: (Windows/MacOS) Windows 11
Hi everyone,
I’m looking to track spherical passive infrared reflective markers, similar to the implementation described in this project for the HoloLens 2: DINO-Unity on GitHub.
Specifically, I’m wondering if it’s possible to use the Depth Sensor to detect the IR reflective material on these markers.
Can the Depth Sensor be leveraged for this purpose? If not, are there alternative approaches you’d recommend for tracking such markers?
I tried the Unity Example projects for WorldCamera and DepthCamera with different configurations without any luck.
Any suggestions, guidance, or resources would be greatly appreciated.
Hi @tkCADS I'm not directly familiar with the project that you shared but you should be able to obtain the raw image from the ToF sensor and use it to detect the reflective IR markers.
Note, we have had feedback that the PixelSensor API does not provide the flexibility that some developers need since you are NOT able to obtain the Raw Image and the Filtered image data at the same time.
Here is some additional information about the Depth Sensor Configurations:
Capability
Description
Range Modes
Long Range (1m - 5m): Operates at 1 or 5 fps
Short Range (0.20m - 1.00m): Operates at 5 or 30 fps (60 fps is supported but can cause overheating when running for extended periods)
Resolution
544 x 480 pixels
Data Options
Developers can query Depth Data or Raw Depth Data
Exposure Adjustment
Developers can adjust the Exposure Time
Depth Range
Developers can clamp the depth range for either mode
Depending on the selected output the following Metadata is available for each frame:
Configuration
Metadata
Depth 32 Output
- Depth Confidence Buffer - Depth Flag Buffer - Exposure Time - Pinhole Camera Model
Raw Depth Output
- Depth Frame Illumination - Exposure Time - Pinhole Camera Model
The Sensor's Position and Rotation can be queried regardless of configuration
Just to clarify, I am looking for a way to achieve an infrared response that allows me to clearly distinguish between the infrared reflective materials and non-reflective materials.
To give you a better idea of what I’m aiming for, I’ve attached an image that illustrates my desired outcome:
I think that would be Depth Frame Illumination, but I'm not familiar with this subject.
If you have a Magic Leap 2 device you can see the latest Unity Example Project , it comes with a Pixel Sensor Example scene. Alternatively we have a Pixel Sensor Section on the Developer Documentation.
I was able to work out the IR response using DepthRaw—thanks for the guidance! However, as you previously mentioned, it’s not possible to access both DepthRaw and Depth32 simultaneously. I have a few questions about potential workarounds:
Is there a way to extract depth data from DepthRaw?
Can the Depth32 buffer provide an IR response, perhaps by leveraging the DepthFrameIllumination option somehow?
I assume switching between the two buffers every frame would not work due to latency—can you confirm?
Are there any alternative methods to obtain both Depth and IR response data at the same time?
I think it would be impossible for you to obtain depth using the DepthRaw image because:
Raw Depth Image : This is the raw depth camera sensor data captured with the depth camera illumination and corresponds to the amount of total light incident on the sensor.
vs
Depth Image: stores the Depth Data from the Depth Camera. Depth is represented in meters
While not on our roadmap, I will put in a feature request to the Voice of Customer team based on your requirements.