Hello,
I'm using GetLatestDepthData from MLDepthCamera API to get the depth information. However, I noticed that the depth data is not really accurate based on what I've measured (I have 2 retroreflective objects with different depth but it gave me same depth value). Is it the known limitations of the depth camera or is there anything I can configure on the API?
Thank you!
Hi @htn002
welcome to the forums.
I would like to know more about your setup physically, this will help me trouble shoot the behavior you are seeing as well as help me give context to any other engineers I happen to contact about this.
For the device, which os build are you using? Also is the device being held stationary in a rig, or worn, for this test?
For the environment, where is this test being done, indoors, outdoors, and what are the lighting conditions in that space? For the targets, what are they? how big are they, and how far away are they from the device and from each other?
Lastly software environment, could you please let me know what version of unity, along with which sdk you are using. If there is a script that you are using for your testing and able to share that would be helpful too.
-Sidney
My setup is indoor, wearing ML2 and looking at a hand held tool with infrared reflective spheres. I exported one frame's raw buffer and intrinsics parameters to process in OpenCV following this post to get the point clouds. I noticed that the distance measurements between the spheres are not really accurate relative to the world
Ok I think the problem may be the spheres. If they are the retroreflective "passive marker spheres" like you would use for multi-camera 3d tracking you will likely get unreliable results. The ML2 uses a Time of Flight Depth Sensor, and ToF sensors can be confused by highly reflective surfaces and especially retroreflective surfaces. Is there a hard requirement to measure using those markers or were they used out of convenience?
Yes, using retroreflective spheres is a must, but the surface material of the spheres can be flexible
Perhaps if you can explain more of your use case then I can think of another way, since the surface being retroreflective is what is likely causing the depth confusion.
The use case I'm working on is to measure the distance between two markers using the depth camera. I believe there may be some confusion arising from the depth camera. Would you mind reviewing the converter code to make sure there's nothing arising from the undistortion part?
def camera_space_converter(img_x, img_y, depth):
# Camera
focal_length = np.array([365.76, 365.76])
principal_point = np.array([265.92, 240.93])
resolution = np.array([544, 480])
# Distortion coefficients
K1, K2, K3 = -0.0865550637245178, -0.0201911050826311, -0.00704800290986896
P1, P2 = 0.000115656432171818, -0.000655817857477814
# Camera to world transformation
cam_to_world_mat = np.array([
[0.25, 0.11, -0.96, -0.28],
[0.01, 0.99, 0.11, -0.40],
[0.97, -0.04, 0.24, 0.17],
[0.00, 0.00, 0.00, 1.00]
])
uv = np.array([img_x, img_y]) / np.array(resolution)
offset_from_center = uv - np.array([0.5, 0.5])
r2 = np.dot(offset_from_center, offset_from_center)
r4 = r2 * r2
r6 = r2 * r4
radial_distortion_correction = offset_from_center * (1 + K1 * r2 + K2 * r4 + K3 * r6)
tangential_distortion_correction_x = 2 * P1 * offset_from_center[0] * offset_from_center[1] + P2 * (r2 + 2 * offset_from_center[0] * offset_from_center[0])
tangential_distortion_correction_y = 2 * P2 * offset_from_center[0] * offset_from_center[1] + P1 * (r2 + 2 * offset_from_center[1] * offset_from_center[1])
tangential_distortion_correction = np.array([tangential_distortion_correction_x, tangential_distortion_correction_y])
corrected_uv = radial_distortion_correction + tangential_distortion_correction + np.array([0.5, 0.5])
camera_point = (corrected_uv * resolution - principal_point) / focal_length
camera_point = np.append(camera_point, 1)
camera_point = camera_point/np.linalg.norm(camera_point) * depth
world_point = cam_to_world_mat @ np.append(camera_point, 1)
return world_point[:3]