Camera intrinsic parameters are not constant

[lakatosp]

Hi there,

I've observed an interesting phenomenon regarding camera intrinsic parameters: they are not constant. If I read through the API them several times during the usage of the camera, I will get slightly different values.

My question would be:

• Why does this happen?
• How the cameras calibration process is done during the production?

Details:

• The affected parameters are the focal distances (fx, fy), the principal point (cx, cy) and the distortion coefficients (k1, k2, p1, p2, k3).
• The affected camera is the video camera (aka VCam, RGB Camera, Picture Camera). Depth camera intrinsic parameters are always the same. I haven't checked the other cameras.
• cx, cy, k1, k2, p1, p2 and k3 can't take any values but one of 16 possible values.
  • I've found 16 different values, but probably I haven't found them all.
  • They are also completely correlated: there are 16 different tuples of 7 values, and cx, cy, k1, k2, p1, p2 and k3 can be together one of this 16 tuple.
  • The possible values of these parameters are evenly distributed over a range.
  • Here is a plot about the co-appearing values of cx and p1:
    • Regarding the missing value see my previous point about not finding all the possible values.
    • We have similar plots for any 2 of this 7 parameters.
      
      
• fx and fy can also take values from range of values.
  • I've found 47 possible 2-tuples of values, but I definitively haven't found them all.
  • They are also completely correlated with each other:
    
    
• Sensor temperature:
  • fx and fy are correlated with sensor temperature, which is understandable
  • The other 7 are NOT correlated with sensor temperature, which begs for the question: If not the sensor temperature, than what modifies them?
  • Also cx, cy, k1, k2, p1, p2 and k3 are NOT correlated with fx and fy:
    
  • But there is a many-to-one correspondence between values of focal length and the other 7 parameter:
    • For any cx value we can have several fx values.
    • But any fx value belongs to exactly 1 of the 16 possible cx values.
    • The fx values belongs to the same cx are quite intermingled, see the plot above.
• Different Magic Leap 2 devices have different tuples of possible values.

Could you please explain what is going on in the background here? Thank you in advance! Péter

[kbabilinski]

Each ML2 device goes through a calibration procedure thus each device would have different intrinsic values. Additionally, we provide per-frame intrinsic that take into account the device temperature, the focus distance to calculate intrinsic so intrinsic would change because of these variables.

Do you see this when using the CV and Main video camera streams?

[lakatosp]

Hi kbabilinski, thanks for the quick reply.

I was not aware of the autofocus function, that explains a lot. Maybe do you have any demo application for the autofocus?

I see this on the RGB camera, but not on the depth camera. And I haven't checked any of the world cameras.

So if I understand correctly, the current, per-frame value of cx, cy, k1, k2, p1, p2 and k3 depends on the autofocus' current state but not on the sensor temperature.

What still puzzles me is that the value of fx and fy is correlated with the temperature, but not really with the other parameters. (See my 3rd image for example.) Does the value of fx and fy depends on both the autofocus' current state and the temperature? If so, is it possible that the temperature has much greater effect on them?

[lakatosp]

Also, can I get from the API the sensor temperature and all the possible values of k1, k2, p1, p2 and k3 at once? It looks like that there are only a manageable number of possible values exists. The use case for the latter would be to pre-calculate the undistortion transformation map during camera startup for all possible values and thus speed up the undistortion significantly.

[kbabilinski]

I can't share the exact details on the distortion algorithm used by our OS but I do know that the Magic Leap 2 supports Androids Camera NDK APIs if you are more familiar with those APIs. Otherwise you can see the ml_camera_metadata header for the MLCamera API documentation.

Regarding the temperature of the sensor and sensor data required to calculate all of the transformations to speed up distortion, This functionality is not available and not currently on our roadmap. However, I will share your request for these values with our voice of customer team.

[lakatosp]

Thank you for sharing it with the voice of customer team! I think it is the only sound way for a real-time undistortion of the RGB camera image. (In the meantime I will collect the possible values and save them to a device-specific file.)

Thanks for the support!