Overview
While the Android Camera2 API provides fields like LENS_INTRINSIC_CALIBRATION and LENS_DISTORTION , device manufacturers may handle these differently. Which is why these specific fields are not populated with data on the Magic Leap 2.
To obtain the intrinsic calibration values of a camera in Unreal using Android, you would typically use the ML_CONTROL_CAMERA_INTRINSICS or ML_CONTROL_MRCAMERA_INTRINSICS tags to query the camera’s intrinsic parameters from the capture result metadata. The provided code snippet demonstrates how to retrieve these values using the Android Camera NDK.
Magic Leap Vendor Tags
The main difference between the two tags, ML_CONTROL_CAMERA_INTRINSICS and ML_CONTROL_MRCAMERA_INTRINSICS, seems to be the specific camera they are querying. The ML_CONTROL_CAMERA_INTRINSICS tag is used for the main or CV (Computer Vision) camera, while the ML_CONTROL_MRCAMERA_INTRINSICS tag is for the MR (Mixed Reality) camera. This distinction is important because the intrinsic parameters might differ between the main/CV and MR cameras due to their different uses.
How to obtain frame intrinsics
The process to obtain the intrinsics based on the example involves the following steps:
- Define a custom structure (
ml_intrinsics) to hold the intrinsic parameters.
- In the
onCaptureCompleted callback function, check which camera is currently being used (MLCamera_Main or MLCamera_CV for the main/CV camera, and any other case for the MR camera).
- Use the appropriate tag (
ML_CONTROL_CAMERA_INTRINSICS or ML_CONTROL_MRCAMERA_INTRINSICS) to query the ACameraMetadata for the intrinsic parameters.
- Retrieve the focus distance using the
ACAMERA_LENS_FOCUS_DISTANCE tag.
- Store the retrieved intrinsic values in the
app->intrinsics_ variable and the focus distance in app->result_focus_distance_.
It’s important to handle any errors that might occur during the metadata retrieval process, as shown in the error-checking code within the function.
Example:
Headers to include:
// Magic Leap Specific
#include "ml_camera_vendor_tags.h" // For Magic Leap vendor-specific tags
// Generic Android
#include <camera/NdkCameraManager.h> // For camera manager functions
#include <camera/NdkCameraDevice.h> // For camera device functions
#include <camera/NdkCameraMetadata.h> // For camera metadata functions
#include <camera/NdkCaptureRequest.h> // For capture request functions
#include <media/NdkImage.h> // For image-related functions
// ... (other includes and code)
Custom Intrinsics Struct
// Struct to contain the Magic Leap Intrinsics
typedef struct ml_intrinsics {
float width; // Image width (in pixels)
float height; // Image height (in pixels)
float fx; // Focal length along the x-axis (in pixels)
float fy; // Focal length along the y-axis (in pixels)
float px; // Principal point x-coordinate (in pixels)
float py; // Principal point y-coordinate (in pixels)
float fov; // Field of view (typically in degrees)
float k1, k2, k3; // Radial distortion coefficients (often follows Brown's distortion model)
float p1, p2; // Tangential distortion coefficients
} ml_intrinsics;
Capture Completed Function
// Callback function for when a capture is completed.
// This function is called after a capture session is completed and processes the resulting metadata to obtain camera intrinsic parameters.
static void onCaptureCompleted(void* context, ACameraCaptureSession* /*session*/,
ACaptureRequest* /*request*/, const ACameraMetadata* result) {
// Cast the context to the specific application type to access its members.
auto app = reinterpret_cast<CameraNdkIntrinsicsApp*>(context);
// Initialize the camera status to OK.
camera_status_t ret = ACAMERA_OK;
// Ensure the application context is not null before proceeding.
if (app == nullptr) return;
// Declare metadata entries for intrinsics and focus distance.
ACameraMetadata_const_entry entry;
ACameraMetadata_const_entry dist_entry;
// Check which camera is currently being used (Main or CV) and attempt to retrieve the relevant intrinsic parameters.
if (app->cur_camera_ == MLCamera_Main || app->cur_camera_ == MLCamera_CV) {
// Attempt to get the intrinsic metadata for the main or CV camera using a predefined tag.
ret = ACameraMetadata_getConstEntry(result,
ML_CONTROL_CAMERA_INTRINSICS,
&entry);
// If retrieval fails, log an error message and return early.
if (ret != ACAMERA_OK) {
ALOGE("get intrinsics metadata for main or cv camera failed\n");
return;
}
} else {
// For other camera types, attempt to retrieve intrinsics using a different predefined tag.
ret = ACameraMetadata_getConstEntry(result,
ML_CONTROL_MRCAMERA_INTRINSICS,
&entry);
// If retrieval fails, log an error message and return early.
if (ret != ACAMERA_OK) {
ALOGE("get intrinsics metadata for mr camera failed\n");
return;
}
}
// Attempt to retrieve the focus distance metadata.
ret = ACameraMetadata_getConstEntry(result,
ACAMERA_LENS_FOCUS_DISTANCE,
&dist_entry);
// If retrieval fails, log an error message and return early.
if (ret != ACAMERA_OK) {
ALOGE("get focus distance metadata failed\n");
return;
}
// Assign the retrieved intrinsics data to the application's intrinsics member.
// The data is cast to the ml_intrinsics struct to match the expected format.
app->intrinsics_ = *(ml_intrinsics *)entry.data.f;
// Assign the first element of the focus distance array to the application's focus distance member.
app->result_focus_distance_ = dist_entry.data.f[0];
}
Resources
Tags used in the example (see ml_camera_vendor_tags.h) for more details:
//...
/*!
\brief ML_CONTROL_CAMERA_INTRINSICS can be used to query the camera intrinsics in Capture Result Metadata.
The tag can be used to query intrinsics for camera.
The tag is available in Capture Result metadata.
The tag has data type float and returns an array of floats which can be interpreted as:
num_intrinsics * {width, height, focal_length_x, focal_length_y, principal_point_x, principal_point_y, field_of_view, k1, k2, p1, p2, k3}.
The name for this tag is "com.amd.control.intrinsics" which can be used with android java camera API.
*/
#define ML_CONTROL_CAMERA_INTRINSICS 0x8000001C
/*!
\brief ML_CONTROL_MRCAMERA_INTRINSICS can be used to query the camera intrinsics in Capture Result Metadata.
The tag can be used to query intrinsics for camera.
The tag is available in Capture Result metadata.
The tag has data type float and returns an array of floats which can be interpreted as:
num_intrinsics * {width, height, focal_length_x, focal_length_y, principal_point_x, principal_point_y, field_of_view, k1, k2, p1, p2, k3}.
The name for this tag is "com.ml.control.intrinsics" which can be used with android java camera API.
*/
#define ML_CONTROL_MRCAMERA_INTRINSICS 0x80000113
//...