How can I get the camera matrix (intrinsic) for RGB and the Depth camera? And are there calibrated extrinsic, like the translation and rotation matrix between the RGB and the Depth?
I’m inching my way towards combining the two into an RGBD image, or at least an RGB and D that align with each other as well as I can get it.
Thanks!
Give us as much detail as possible regarding the issue you’re experiencing:
Unity Editor version: 6000 ML2 OS version: 1.12 Unity SDK version:2.6 Host OS: (Windows/MacOS) windows, but running on the ML2
Error messages from logs (syntax-highlighting is supported via Markdown):
Never met a headset that made this stuff easy How about this question. I ended up having to use MLCamera to get my RGB8888 data, and then Pixel sensor to get my depth data. That’s finally all working. But… I have to use PixelSensorFeature.GetSensorPose(SensorId); for the pixel sensor because getting it from the frame just never works. Then in MLCVCamera I do MLCVCamera.GetFramePose(mlTime, out mlPose).IsOk.
And with the headset still, the poses are too different to be correct. Like the translation baseline is 120mm. So I’m assuming these two things do not return the same pose? Docs seem to be clear the ML camera pose is the pose of the camera relative to the world origin, but GetSensorPose just says “pose of sensor”.
You’re correct, those two APIs do not return the same reference frame by default.
MLCVCamera.GetFramePose() returns the pose in Unbounded tracking space (MLSDK default), which is anchored to the world origin defined by the system.
PixelSensorFeature.GetSensorPose() returns the pose in the tracking origin set in your XR Origin (usually Device or Floor space in Unity). This is why you see the large offset.
If you need them to align:
1 Make sure your tracking space is set to Unbounded as mentioned in the OpenXR MLCamera doc.
2. To query based on the frame timestamp, See this SDK modification example:
3. You also might want to verify the pose is not returning (0,0,0) before processing.
This way, both ML Camera and Pixel Sensor poses will report from the same origin.
Is there a way to get transform between the two spaces? I ask because I’m capturing and saving the mesh as a ply file in Unity world space (using localToWorldMatrix). But I’m stuck with the unbounded pose from MLCamera. And I’m using MLCamera because it was the only way I found to give me RGB data instead of a jpeg stream.
For a test I used the PixelSensor pose from the depth camera and the results are so close! Just need to get the right pose for RGB
When you use the Unbounded Reference Space, the Pixel Sensor should provide it’s pose in the same manner.
Make sure you have the reference Space Feature enabled in the Unity Settings and that your script toggles the unbounded space before the capture begins.
How about this question. I ended up having to use MLCamera to get my RGB8888 data, and then Pixel sensor to get my depth data. That’s finally all working. But… I have to use PixelSensorFeature.GetSensorPose(SensorId); for the pixel sensor because getting it from the frame just never works. Then in MLCVCamera I do MLCVCamera.GetFramePose(mlTime, out mlPose).IsOk.