Questions about the new eye tracker feature

Unity Development
,
1

Give us as much detail as possible regarding the issue you're experiencing:

Unity Editor version: 2022.3.42f1
ML2 OS version: 1.9.0
Unity SDK version: 2.4.0
Host OS: Windows

I have been testing the newly implemented eye tracking features in OpenXR. Overall it looked nice, however there are some issues I encountered when I look at the returns and hopefully you might be able to answer a few of them.

  1. Velocity, Direction, Amplitude in GazeBehavior MetaData. It seems that "NaN" is returned whenever there's a fixation or blink (or unknown). Is this expected behavior? And what does it mean to have "direction" represented as a float value? The velocity in saccades or pursuits are still valuable, but it does not align with what I got using combined gaze readings from OpenXR gaze interactions with a window size of 3 (~30ms interval, I think it shouldn't be larger). Is there any chance that you may talk about the underlying algorithm?
  2. The gaze behavior OnsetTime seems to be pretty far away from the current timestamp. For example, here's a typical fixation event detected, left column being current gaze event and right being onset time. It seems to be >60s apart? Or is there any conversion that needs to be applied to the onset time?

image

  1. I was a bit confused by the "pursuit" labels, as when I fixated on a moving object it often labeled the event as "fixation", but when I moved my head and remain fixated on a single point then it's more likely recognized as "pursuit". Is there a specific definition of what is being labeled as "pursuit"? I think the current label probably would align more with the concept of VOR (vestibulo-ocular reflex).

I understand that some of these questions might be hard for you to answer (revealing underlying algorithm), but would really appreciate some help on this so that we (as a research team) can leverage the eye tracking capabilities more on Magic Leap 2.

BTW, I also found that when running Unity apps, the streaming function on either Magic leap hub 2 or 3 cannot record the real world content. Is this related with new security protocols implemented on newer OS versions? Will appreciate help on this as well.

Thanks in advance!

Best,
Zhehan

2

We will take a look at the eye tracking questions.

Regarding the MLHub streaming issue, please make sure that the segmented dimmer feature is disabled and that the display mode is set to Additive.

Alternatively, you will need to make sure that the alpha channel on the image that is being sent to the Magic Leap 2 to render is accurate. By disabling HDR on your camera and Render Pipeline settings. Additionally, make sure that post processing is also disabled.

3

Regarding Some additional information about the API, here is the OpenXR Spec which includes details about the states and some defenitions:

4

I was able to track down the information regarding the Eye Tracker API:

1: Velocity, Direction, Amplitude in GazeBehavior MetaData. It seems that "NaN" is returned whenever there's a fixation or blink (or unknown). Is this expected behavior?

Answer: Having nulls for velocity , amplitude and direction when we are in blink, unknown or fixation state is expected and occurs when we do not have any motion data from either the head or eyes.

2: What does it mean to have "direction" represented as a float value?

Answer: The angle field in XrEyeTrackerGazeBehaviorMetaDataML represents the direction. It's specified in radians, ranging from 0 to 360, with 0 being at the 3 o'clock position and increasing counter-clockwise.

3: The velocity in saccades or pursuits are still valuable, but it does not align with what I got using combined gaze readings from OpenXR gaze interactions with a window size of 3 (~30ms interval, I think it shouldn't be larger). (underlying algorithm?)

Answer: The velocities in Gaze Behaviors are based on 2D computations (e.g., just looking at the pupil center on the image) so they are more of a relative scale versus the 3D metric values available from the gaze API.

4: The gaze behavior OnsetTime seems to be pretty far away from the current timestamp. For example, here's a typical fixation event detected, left column being current gaze event and right being onset time. It seems to be >60s apart? Is there any conversion that needs to be applied to the onset time?

Answer: We are working on a fix for the timestamps and are aiming to have the fix implemented in the next OS release.

5: I was a bit confused by the "pursuit" labels, as when I fixated on a moving object it often labeled the event as "fixation", but when I moved my head and remain fixated on a single point then it's more likely recognized as "pursuit". Is there a specific definition of what is being labeled as "pursuit"? I think the current label probably would align more with the concept of VOR (vestibulo-ocular reflex).

Answer: When we start moving head, the pupils start also moving and we are not longer in fixation state. The specification defines pursuit as "User is pursuing, eye velocity is low but nonzero." When you move your head while fixated on a stationary point, your eyes make larger, more pronounced movements to maintain fixation. This might cross the velocity threshold into the pursuit category.

5

Many thanks for your detailed reply!

I haven't have the chance to test the sigmented dimmer feature/ additive display mode but I will run a quick test next week.

It makes sense if the velocity values are obtained from eye images and thanks for looking into the timestamp issue!

On the other hand, the smooth pursuit issue still bothers me a bit. I understand when pursuit is labeled for VOR, but my question is more about why when there's actually a smooth pursuit towards a moving object with the head being stationary, the label is often "Fixation"? Is it that, based on your observation, VOR tends to lead to higher velocity, in the way that you are measuring it, that smooth pursuit wouldn't lead to? It seems to me that, if the objected being fixated has not moved (even if the head is moved), it's actually more reasonable to label it as a fixation, instead of the opposite way.

Thanks again!

6

Thank you for your feedback. I have submitted your feedback to the voice of customer team. One thing to note, the eye tracker measures the rotation of the eye in the head. It can't discern (without using auxilliary information like the IMUs) whether it is the eye rotating in the head or the head rotating around the eye. The eye tracker is based on the idea that the movements are predominantly eye movements with head rotation being a lesser part.