Give us as much detail as possible regarding the issue you're experiencing.
Unity Editor version: 2022.3 ML2 OS version:c 1.4.0 MLSDK version: 1.4.0 Host OS: (Windows/MacOS) Windows 11 23H2
Error messages from logs (syntax-highlighting is supported via Markdown):
Good Morning everyone,
i'm building an application through unity for surgeons. We can achieve good marker accuracy when using integrated APIs, very well done. But it is impossible to track robustly with sub-millimeter accuracy, which is strictly needed when superimposing patient 3d model to the real patient.
What i'm looking for is a way to use markers composed of more than one (like a charuco board, a diamond aruco and so on..).
What i've tried is the following:
-OpenCV integration, a user from there has succesfully used openCV Android release when doing native development (i'm using unity because my work has to be left to my lab and usable from everyone, so doing only code it's not good for me, but maybe i can make something like an interface).
-Using the single marker poses in order to compute a mean value, but this is very far from the optimal solution since the Perspective-n-Point algorithm works better if multiple data are given to it (like marker corner coordinates).
Is there a way to modify the ML_MarkerTracker API? I'm pretty sure that in Magic Leap a lot of stuff are done as the literature suggest, so maybe i can achieve what i need by directly modifying the API source code. Where can i find it (if it is findable)?
Welcome to the MagicLeap forums! I'm sorry, the native MagicLeap marker tracking C apis are not open source.
If you're using Unity, you can view and modify the C# wrapper code in the MagicLeap Unity sdk that would ultimately wrap the ML native apis. In a Unity project, you can take a look at the source code in this folder to see how the Unity apis are implemented- Packages\com.magicleap.unitysdk\Runtime\APIs\MarkerTracker
If you'd like to use OpenCV or a custom solution for marker/chessboard/corner detection, camera images are available through the ML apis for Unity and native. If you want to use OpenCV in Unity, you'll need to find a c# wrapper or write your own for the functions that you would like to use.
To clarify requirements, do you need to be able to track the movement of markers from frame to frame, or just use them to establish an initial frame of reference? After detecting a marker once, you could thereafter just rely on headpose (in a reasonably small workspace) or spatial anchors to localize content. Are you trying to use them for more than just localizing content for rendering? I'm not sure if sub-millimeter content movement would be visually perceptible in the ML display.
First of all, thank you very much for the (very) fast reply.
Secondly, i'm learning day by day how Unity and Magic Leap 2 work, since i've started my development journey 2 months ago (anyway i'm able to code, so not so bad :D).
What i need is "freedom", meaning that i've to try "easily" different things, the task is to place accurately an hologram with respect to a patient' body, this can be done in two way: one in continuos tracking and one for placement tracking and then rely on the integrated SLAM algorithm to maintain the headset position aligned with the ambient. Another stuff is also to track continuosly instrumentation and tools, because maybe i need to align something with respect to the patient. The problems which arises are several. The PnP alorithm for pose estimation works better with multiple markers, so i would modify it in order to accept more than one marker by knowing the real relative positions. Secondly i need access to estimated corner positions in order to try implementing an algorithm to estimate the accuracy error of hologram placement (ie: reprojection error isn't always a good measure to quantify error placement. Current 2D navigator have an accuracy in the order of <2mm and <2°, we are often able to do equal and sometimes even better, but i want to try to improve it more since the ambient awareness of the Magic Leap 2 seems very stable with a meshing vertex confidence >0.75/0.8. Here arises another question: how to measure how much good the magic leap is aware of the ambient? I've thought about 2 ways (this is another question, let me know if it is better to make another thread): Space confidence, but related only to recognized space created through the SpacesApp, or vertex confidence obtained through the MeshingSubsystem? I'm not able to always pre-register a space.
At the end, and this is another different question, how to share locally with another magic leap 2 the same world coordinates? In order to show hologram in the same position also for the second magic leap 2 (i can share spaces, but i'm not sure they are the right thing for me). The reason is that i would want a way to share without internet access (but a local wlan yes) content data between users, or also to make accurate recording since in the capture there is a slight superimposition offset (now is slight after the last update, before was very much).
Thank you very much for any information,
if needed i can make several thread for my various question.
See you.
Regarding using Markers to Improve Tracking : Currently, you cannot feed marker data or known points to improve the Magic Leap's localization, however we have created a feedback ticket for this functionality and sent it to our Voice of Customer team. I also recommend exploring using OpenCV for Unity if you are unfamiliar with Native. It includes a lot of the same functionality as the Native Library. Note, the OpenCV for Unity package is not configured for Magic Leap 2 out of the box. You will have to obtain the images yourself and can use the Examples in the OpenCV unity package to see how to perform the OpenCV functions.
Regarding Mesh and Tracking Confidence : The Magic Leap Unity Example project demonstrates how to use the meshing subsystem which includes the option to provide vertex confidence. However, this confidence measures the error of the mesh and depth data rather than the localization. For tracking confidence you can poll the Head Tracking state as shown in this example : Handling Tracking Loss Events | MagicLeap Developer Documentation
Regarding Sharing Maps: The spaces application allows you to export and import Spaces at runtime. Meaning that if the space was mapped on one device, you can share it to another device. See the MLSpace Examples on our developer forum for more information : MLSpace Examples | MagicLeap Developer Documentation.
Regarding Shared Origin: If you cannot pre-map the space or want to you an alternative method, I recommend using a marker to defined an origin for your content. While the headsets will still have deferent origins and global positions, the content can be placed relative to the marker or anchor. A simple implementation of this would be to make the Virtual object a child of the transform that represents the marker and sending the virtual object's local position, rotation values rather than the global ones.
Multiplayer examples : You can see the the Magic Leap Photon Fusion Example or the Wifi Direct Shared Experience Example . While these demos show colocation using Photon or Wifi Direct you can implement similar logic regardless of the Network solution. Which means that you can use something like Unity's Netcode for GameObjects if you find it more appropriate.
Last question about the topic regarding "cameras".
I've tried the SDK example scene named "WorldCameras" and also the ones called CameraCapture and CameraCV. I'm trying to understand the camera intrinsics parameters (i'm new in this field). I need mainly to understand two things at the moment:
the measurement units (radial and tangential distortion coeffiecients are all in the same unit for every camera kind? I read that some are in pixel and others are normalized, but normalized in which way?
are the camera intrinsics changed at runtime by the OS? Because i can see focal lenghts slightly change over time, but i've not totally catched if other parameters changes too.
Thank you very much!
And have a good weekend if someone read here before monday (i will return here on that day).
Here is some information regarding the intrinsic from the C++ World Camera API
/*!
\brief World camera intrinsic parameters.
\apilevel 23
*/
typedef struct MLWorldCameraIntrinsics {
/*! Camera width. */
uint32_t width;
/*! Camera height. */
uint32_t height;
/*! Camera focal length. */
MLVec2f focal_length;
/*! Camera principal point. */
MLVec2f principal_point;
/*! Field of view in degrees. */
float fov;
/*!
\brief Radial distortion vector.
The radial distortion co-efficients are in the following order: [k1, k2, k3, k4].
*/
double radial_distortion[MLWorldCameraIntrinsics_MaxRadialDistortionCoefficients];
/*!
\brief Tangential distortion vector.
The tangential distortion co-efficients are in the following order: [p1, p2].
*/
double tangential_distortion[MLWorldCameraIntrinsics_MaxTangentialDistortionCoefficients];
} MLWorldCameraIntrinsics;
The focal length of a lens can also change slightly when the lens is focused at different distances.
thank you for your response.
-I had already seen the documentation but i cannot understand if is there a sort of "online camera calibration" about intrinsics parameters and what it's their measurement unit (they should be without measurement unit since they are coefficient, but they are normalized i'm not sure in which way.
-Moreover it could be useful to know if it is possible to lock auto-focus, both for the rgb or world cameras. Since for computer vision tasks it is an important parameter to consider (i'm trying to achieve submillimeter tracking accuracy + hologram overlays).
-Concluding, i would be interested even in understand how to manage stereo vision. Is there a way to capture at the same time image from more than one world camera? Because i know i've timestamps, but before to try i need to know if is there the possibility to take them at the same time.
Thank you very much, i think that Magic Leap 2 could be a very interesting device in a lot of areas if more information are provided about the hardware and the Apis. (for example, i cannot find, but maybe i'm blind ahahahah, the hardware specs of camera, like sensors used, they physical parameters and so on..)
Regarding autofocus, the camera autofocus cannot be locked because these are the same cameras that are used for head tracking. This means that any changes to the camera could result in unstable tracking.
Regarding capturing multiple cameras at the same time. The API allows you to poll the images from the world cameras which are returned as an array with a given timestamp. These images can be used together to perform custom CV related tasks although they would need to be undistorted before being processed.
Regarding the intrinsic measurement, I will need to double check but I was able to use the OpenCV Fisheye distortion model and passed in the tangential coefficients to undistort the images
To undistort the image you will need to use The model is opencv fisheye OpenCV: Fisheye camera model model and with tangential terms.
// Magic Leap two step fisheye distortion model
// k1, k2, k3, k4 = radial_dist_coeffs
// p1, p2 = tangential_dist_coeffs
//
// R = sqrt(x^2 + y^2)
// theta = arctan(R)
// rd = (theta + k1 * theta^3 + k2 * theta^5 + k3 * theta^7 + k4 * theta^9) / R
// x = x * rd
// y = y * rd
// r = x^2 + y^2
// x_dist = x + 2 * p1 * x * y + p2 * (r + 2 * x^2)
// y_dist = y + p1 * (r + 2 * yi^2) + 2 * p2 * x * y
x and y in above snippet are in normalized image coordinates. They correspond to a and b in opencv's documentation
Thank you very much @kbabilinski , i missed the fact that they are fisheye cameras, for this reason i didn't understood the forth radial coefficient. I will try better probably the next year.
In the meanwhile, happy end of december!
I will try