I was unable to find any documentation on the ability to access the IMU data on the ML2. Can I read real-time data from the accelerometer, gyroscope, and magnetometer? Does this require some "developer work arounds" like the HoloLens 2 did?
3 Likes
Hi @mattycorbett For Magic Leap 2, IMU data is exposed via Android Sensor Management APIs.
You can learn more here
1 Like
Thank you! Excuse my ignorance here, but I am coming over from developing for the HoloLens and HoloLens 2. Can these APIs be used in Unity?
2 Likes
Sorry I can't answer for the Unity side of things, but I also was just looking into reading IMU data from the headset as well on the Android Native API side, so for anyone who's interested in that, this is what I found.
First, you need to modify your CMakeLists.txt to add the include directories for the NDK
include_directories(C:/Users/<your user>/AppData/Local/Android/Sdk/ndk/25.0.8775105/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/include/android)
Then in your project, include the NDK's <sensor.h>
Add class member variables for an ASensorManager* and an ASensorEventQueue*
ASensorManager* mSensorManager = nullptr;
ASensorEventQueue* mSensorEventQueue = nullptr;
In the activity constructor I call
// get sensor manager instance
mSensorManager = ASensorManager_getInstanceForPackage("com.magicleap.capi.sample.camera_preview"); // your package name here (not sure why it's important, and doesn't seem to matter even if name doesn't actually exist
// get all the available sensors
ASensorList sensorList;
int numSensors = ASensorManager_getSensorList(mSensorManager, &sensorList);
// print available sensors
for (int i = 0; i < numSensors; i++)
{
ASensorRef sensor = sensorList[i];
ALOGE("%s - %s", ASensor_getName(sensor), ASensor_getStringType(sensor));
}
// Create an event queue that polls for sensor updates
int looperID = 3; // random value copied from example code, not sure its significance
mSensorEventQueue = ASensorManager_createEventQueue(mSensorManager, ALooper_forThread(), looperID,
onSensorChanged, // the static function callback that gets called on sensor update
this); // void* user data that gets passed to callback
// Example sensor:
// I want to track the gyroscope data
// Use default gyro sensor
auto *sensor = const_cast<ASensor *>(ASensorManager_getDefaultSensor(mSensorManager, ASENSOR_TYPE_GYROSCOPE));
// enable the gyro sensor
auto status = ASensorEventQueue_enableSensor(mSensorEventQueue, sensor);
// set how often to poll the sensor in micro-seconds
int usec = 100000;
status = ASensorEventQueue_setEventRate(mSensorEventQueue, sensor, usec);
And here is the callback called on sensor updates
/** callback called by ASensorManager_createEventQueue() */
static int onSensorChanged(int fd, int events, void* data)
{
// get the app passed as user data via void*
CameraPreviewApp* this_app = (CameraPreviewApp*)data;
// get all sensor events
ASensorEvent event;
while (ASensorEventQueue_getEvents(this_app->mSensorEventQueue, &event, 1) > 0)
{
// do what you will with said events
ASensorVector gyro = event.gyro;
ALOGE("Gyro Data: %lf, %lf, %lf", gyro.x, gyro.y, gyro.z);
}
return 1;
}
And here's the list of sensors I get:
E/com.magicleap.capi.sample.camera_preview: Headset Right Accelerometer Sensor - android.sensor.accelerometer
E/com.magicleap.capi.sample.camera_preview: Headset Right Gyroscope Sensor - android.sensor.gyroscope
E/com.magicleap.capi.sample.camera_preview: Headset Left Accelerometer Sensor - android.sensor.accelerometer
E/com.magicleap.capi.sample.camera_preview: Headset Left Gyroscope Sensor - android.sensor.gyroscope
E/com.magicleap.capi.sample.camera_preview: Compute Pack Accelerometer Sensor - android.sensor.accelerometer
E/com.magicleap.capi.sample.camera_preview: Compute Pack Gyroscope Sensor - android.sensor.gyroscope
E/com.magicleap.capi.sample.camera_preview: Ambient Light Sensor - android.sensor.light
E/com.magicleap.capi.sample.camera_preview: Game Rotation Vector Sensor - android.sensor.game_rotation_vector
E/com.magicleap.capi.sample.camera_preview: Gravity Sensor - android.sensor.gravity
And some sample gyro values I get while sitting still
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.178970, 0.111856, 0.086289
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.177905, 0.110258, 0.086822
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.176307, 0.108128, 0.087354
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.174709, 0.106530, 0.087354
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.173111, 0.104399, 0.086822
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.171513, 0.102269, 0.086289
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.170448, 0.100671, 0.085224
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.168850, 0.099073, 0.084691
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.167784, 0.097475, 0.084158
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.166719, 0.095877, 0.084158
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.165654, 0.094279, 0.084158
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.164056, 0.092681, 0.084691
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.162458, 0.091616, 0.084691
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.160860, 0.090018, 0.084158
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.159262, 0.088420, 0.083626
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.158197, 0.086822, 0.083093
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.157131, 0.085224, 0.082561
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.156066, 0.083093, 0.082028
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.155001, 0.081495, 0.082028
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.153403, 0.079897, 0.081495
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.151272, 0.078832, 0.081495
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.149142, 0.077234, 0.081495
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.147011, 0.075636, 0.081495
E/com.magicleap.capi.sample.camera_preview: Gyro Data: -0.145413, 0.074038, 0.080963
I'm no expert in this by any means, and most of this I learned an hour ago by referencing this github repo that writes a shallow c++ wrapper around the api, but this should be enough to get started. Once again, sorry I can't speak to the Unity side, but this should help anyone working in the native API.
2 Likes
Does this/these features require the developer pro model? Is the answer the same with eye gaze apis?
1 Like
Yes @mattycorbett These APIs can be used in Unity.
1 Like
@mattycorbett You can work with these APIs with the Magic Leap 2 base edition. You do not need a Developer Pro license to do so. You can see a breakdown of the differences between the editions on our Where to Buy page.
1 Like
Do you have a link to the Unity APIs? Struggling to find them....
@mattycorbett At this time we have not documented this process because we use the standard Android sensor API. I recommend asking the question on Unity’s forums since it IMU data collection is not specific to our platform. You can create a native plugin to read the sensor data and pass it into your Unity application or search the Unity asset store for a precompiled version. Please let us know if you run into any specific issues.
Hi, I have encountered a similar problem. I am reading the IMU data using a native API, but it appears that there is a buffer to store 24 sets of IMU data. I can read all of them, but afterward, I have to wait for 24ms (I have set the rate to 1024Hz) before I can perform the next read due to this buffer limitation. Is there any method to read the IMU data in real-time? By the way can I poll sensor data by 1000us with "
ASensorEventQueue_setEventRate "
According to the Android NDK documentation, the function ASensorEventQueue_setEventRate sets the delivery rate of events in microseconds for the given sensor. This means that you can specify how often you want to receive sensor data from the device. However, this function does not guarantee that you will get the data in real-time, as there might be some latency or delay depending on the device and the sensor.
One possible way to reduce the latency and get more accurate sensor data is to use the sensor direct report feature, which was introduced in Android 7.0 (API level 24). This feature allows you to create a direct channel between the sensor and a shared memory buffer, bypassing the system sensor service. You can then read the sensor data from the shared memory buffer at any time, without waiting for events from the sensor service.
You may also want to consider the following: The app needs to request HIGH_SAMPLING_RATE_SENSORS permission to access sensor data at a higher rate than 1000Hz. With this permission enabled the application should be able to obtain the IMU data at a higher speed ~5000hz.
It appears that Magic Leap 2 does not support sensor direct reporting. Could you kindly provide an example of code? We attempted to use accSensory?.isDirectChannelTypeSupported(SensorDirectChannel.TYPE_MEMORY_FILE) , but it returned a failure java.io.UncheckedIOException: java.io.IOException: create HardwareBuffer direct channel failed -12.
Device Information:
Android 10.0, API Level 29
Magic Leap2：V1.4.0-dev2/V1.2.1/V1.1.0
Do you mind trying to use the SensorDirectChannel.TYPE_HARDWARE_BUFFER instead?
I will try to track down a sample that uses direct sensor direct report, historically our internal tests only use the callback event since this option is the most common. But I will try to get more information.
@kbabilinski Thanks. We attempted various methods, but they proved ineffective. Additionally, we implemented the callback event, but encountered several issues, including:
- Repetitive data, as indicated by the highlighted area in the image.
- Inconsistencies in time steps between the system time and the IMU data. While polling IMU data, we observed the system time jumping from 43 577ms to 43 599ms (about 22ms), whereas the IMU timestamp only changed from 1 939 485.847ms to 1 939 486.824ms (about 1ms).
Interesting, when using the (ASensorManager_createEventQueue) and setting event rate to 1000hz (ASensorEventQueue_setEventRate) I was able to obtain data at ~ 1000hz . Are you testing this in java, NDK or Unity ?
We acquire data through the native C API and perform tests within the NDK. The data rate is indeed 1000Hz, which is correct. However, as I mentioned earlier, I encounter a delay when polling the IMU data in real-time; I have to wait for 22 ms before the next polling, as depicted in the image.