RGB-D and pose capture

Getting Started
1

I’m just getting started with my ML2, and my goal is to capture RGB-D and pose at as close to the same time as possible. Is there an example of this anywhere? From reading I see I’ll probably have to capture RGB then depth and pose and I’ll need to figure out the extrinsics between the RGB and depth sensor or those are known / provided somewhere?

Thanks!

2

Correct, we do not provide a single API to obtain the RGB and Depth camera streams. Instead you will need to use the ML Camera API and the Pixel Sensor API to obtain the individual RGB and Depth frames. Each of the frames contain the data that you mentioned.

Note that the Depth and RGB Cameras are not synchronized. So your application would need to try to synchronize them / find the closest pairs.

Generally you may want to consider the following flow:

  1. Obtain the RGB and Depth frames
    i. Store the Texture, Pose, Intrinsic and timestamps
  2. Track a few frames to find the frames that best match
  3. When a pair is found within a given tolerance, process the data to align the frames.

This is a non-trivial task. If the scene is static, you may find it easier to use the Meshing API. Then capture the depth from the generated mesh rather than using the depth directly from the sensor.

We have both MLCamera and Pixel Sensor API samples on our developer portal: