# Questions about camera

**URL:** <https://forum.magicleap.cloud/t/questions-about-camera/3201>\
**Category:** General\
**Tags:** development\
**Created:** [February 21, 2024, 9:01am UTC](https://forum.magicleap.cloud/t/questions-about-camera/3201 "2024-02-21T09:01:23Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![bjinsu](https://avatars.discourse-cdn.com/v4/letter/b/a88e57/32.png) [@bjinsu](https://forum.magicleap.cloud/u/bjinsu)\
**Post date:** [February 21, 2024, 9:01am UTC](https://forum.magicleap.cloud/t/questions-about-camera/3201/1 "2024-02-21T09:01:23Z")

</div>

hello. I am developing a Magic Leap app using Unreal.

Currently I need the camera intrinsic and distortion values ​​for the main camera that takes the picture.

Last time I asked a question, I was told to use the Android API to get the camera intrinsic value.

I am using Android camera2 API and was able to obtain TotalCaptureResult obtained through onCaptureCompleted method in CameraCaptureSession.CaptureCallback.

However, when accessed using the LENS\_INTRINSIC\_CALIBRATION field and LENS\_DISTORTION, both return null values.

When I use the other fields of TotalCaptureResult, LENS\_FOCAL\_LENGTH and LENS\_FOCUS\_DISTANCE, I can get the corresponding values.

I wonder if it is correct for Magic Leap to provide LENS\_INTRINSIC\_CALIBRATION and LENS\_DISTORTION values ​​to Android OS.

When I used the Unity API, I was able to get the camera intrinsic value, and I want to get the same value using the Android API.

I need help. thank you

---

<div class="post-metadata">

**Author:** ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)\
**Post date:** [February 22, 2024, 12:05am UTC](https://forum.magicleap.cloud/t/questions-about-camera/3201/2 "2024-02-22T00:05:26Z")

</div>

## Overview

While the Android Camera2 API provides fields like `LENS_INTRINSIC_CALIBRATION` and `LENS_DISTORTION` , device manufacturers may handle these differently. Which is why these specific fields are not populated with data on the Magic Leap 2.

To obtain the intrinsic calibration values of a camera in Unreal using Android, you would typically use the `ML_CONTROL_CAMERA_INTRINSICS` or `ML_CONTROL_MRCAMERA_INTRINSICS` tags to query the camera’s intrinsic parameters from the capture result metadata. The provided code snippet demonstrates how to retrieve these values using the Android Camera NDK.

## Magic Leap Vendor Tags

The main difference between the two tags, `ML_CONTROL_CAMERA_INTRINSICS` and `ML_CONTROL_MRCAMERA_INTRINSICS`, seems to be the specific camera they are querying. The `ML_CONTROL_CAMERA_INTRINSICS` tag is used for the main or CV (Computer Vision) camera, while the `ML_CONTROL_MRCAMERA_INTRINSICS` tag is for the MR (Mixed Reality) camera. This distinction is important because the intrinsic parameters might differ between the main/CV and MR cameras due to their different uses.

## How to obtain frame intrinsics

The process to obtain the intrinsics based on the example involves the following steps:

1. Define a custom structure (`ml_intrinsics`) to hold the intrinsic parameters.
2. In the `onCaptureCompleted` callback function, check which camera is currently being used (`MLCamera_Main` or `MLCamera_CV` for the main/CV camera, and any other case for the MR camera).
3. Use the appropriate tag (`ML_CONTROL_CAMERA_INTRINSICS` or `ML_CONTROL_MRCAMERA_INTRINSICS`) to query the ACameraMetadata for the intrinsic parameters.
4. Retrieve the focus distance using the `ACAMERA_LENS_FOCUS_DISTANCE` tag.
5. Store the retrieved intrinsic values in the `app->intrinsics_` variable and the focus distance in `app->result_focus_distance_`.

It’s important to handle any errors that might occur during the metadata retrieval process, as shown in the error-checking code within the function.

## Example:

### Headers to include:

```cpp
// Magic Leap Specific
#include "ml_camera_vendor_tags.h" // For Magic Leap vendor-specific tags

// Generic Android
#include <camera/NdkCameraManager.h> // For camera manager functions
#include <camera/NdkCameraDevice.h> // For camera device functions
#include <camera/NdkCameraMetadata.h> // For camera metadata functions
#include <camera/NdkCaptureRequest.h> // For capture request functions
#include <media/NdkImage.h> // For image-related functions

// ... (other includes and code)

```

### Custom Intrinsics Struct

```cpp
// Struct to contain the Magic Leap Intrinsics
typedef struct ml_intrinsics {
  float width; // Image width (in pixels)
  float height; // Image height (in pixels)
  float fx; // Focal length along the x-axis (in pixels)
  float fy; // Focal length along the y-axis (in pixels)
  float px; // Principal point x-coordinate (in pixels)
  float py; // Principal point y-coordinate (in pixels)
  float fov; // Field of view (typically in degrees) 
  float k1, k2, k3; // Radial distortion coefficients (often follows Brown's distortion model)
  float p1, p2; // Tangential distortion coefficients
} ml_intrinsics;

```

### Capture Completed Function

```cpp

// Callback function for when a capture is completed.
// This function is called after a capture session is completed and processes the resulting metadata to obtain camera intrinsic parameters.
static void onCaptureCompleted(void* context, ACameraCaptureSession* /*session*/,
                               ACaptureRequest* /*request*/, const ACameraMetadata* result) {
  // Cast the context to the specific application type to access its members.
  auto app = reinterpret_cast<CameraNdkIntrinsicsApp*>(context);
  // Initialize the camera status to OK.
  camera_status_t ret = ACAMERA_OK;

  // Ensure the application context is not null before proceeding.
  if (app == nullptr) return;

  // Declare metadata entries for intrinsics and focus distance.
  ACameraMetadata_const_entry entry;
  ACameraMetadata_const_entry dist_entry;

  // Check which camera is currently being used (Main or CV) and attempt to retrieve the relevant intrinsic parameters.
  if (app->cur_camera_ == MLCamera_Main || app->cur_camera_ == MLCamera_CV) {
    // Attempt to get the intrinsic metadata for the main or CV camera using a predefined tag.
    ret = ACameraMetadata_getConstEntry(result,
                                        ML_CONTROL_CAMERA_INTRINSICS,
                                        &entry);
    // If retrieval fails, log an error message and return early.
    if (ret != ACAMERA_OK) {
      ALOGE("get intrinsics metadata for main or cv camera failed\n");
      return;
    }
  } else {
    // For other camera types, attempt to retrieve intrinsics using a different predefined tag.
    ret = ACameraMetadata_getConstEntry(result,
                                        ML_CONTROL_MRCAMERA_INTRINSICS,
                                        &entry);
    // If retrieval fails, log an error message and return early.
    if (ret != ACAMERA_OK) {
      ALOGE("get intrinsics metadata for mr camera failed\n");
      return;
    }
  }

  // Attempt to retrieve the focus distance metadata.
  ret = ACameraMetadata_getConstEntry(result,
                                      ACAMERA_LENS_FOCUS_DISTANCE,
                                      &dist_entry);
  // If retrieval fails, log an error message and return early.
  if (ret != ACAMERA_OK) {
    ALOGE("get focus distance metadata failed\n");
    return;
  }

  // Assign the retrieved intrinsics data to the application's intrinsics member.
  // The data is cast to the ml_intrinsics struct to match the expected format.
  app->intrinsics_ = *(ml_intrinsics *)entry.data.f;
  // Assign the first element of the focus distance array to the application's focus distance member.
  app->result_focus_distance_ = dist_entry.data.f[0];
}

```

### Resources

Tags used in the example (see `ml_camera_vendor_tags.h`) for more details:

```auto
//...

/*!
  \brief ML_CONTROL_CAMERA_INTRINSICS can be used to query the camera intrinsics in Capture Result Metadata.
         The tag can be used to query intrinsics for camera.
         The tag is available in Capture Result metadata.
         The tag has data type float and returns an array of floats which can be interpreted as:
         num_intrinsics * {width, height, focal_length_x, focal_length_y, principal_point_x, principal_point_y, field_of_view, k1, k2, p1, p2, k3}.
         The name for this tag is "com.amd.control.intrinsics" which can be used with android java camera API.
*/
#define ML_CONTROL_CAMERA_INTRINSICS 0x8000001C

/*!
  \brief ML_CONTROL_MRCAMERA_INTRINSICS can be used to query the camera intrinsics in Capture Result Metadata.
         The tag can be used to query intrinsics for camera.
         The tag is available in Capture Result metadata.
         The tag has data type float and returns an array of floats which can be interpreted as:
         num_intrinsics * {width, height, focal_length_x, focal_length_y, principal_point_x, principal_point_y, field_of_view, k1, k2, p1, p2, k3}.
         The name for this tag is "com.ml.control.intrinsics" which can be used with android java camera API.
*/
#define ML_CONTROL_MRCAMERA_INTRINSICS 0x80000113

//...

```
