Pupil tracking lost

Unity Development
1

Dear community,

We are reading pupil size from the ML2 goggle, and in some rare cases, pupil tracking is lost, and the eye camera images appear to be mis-cropped, as shown below. Does anyone have any idea what might be causing this?

Screenshot 2025-02-18 at 10.03.29 AM
Screenshot 2025-02-18 at 10.03.29 AM1920×784 52 KB

These are the left and right temple eye images from the same acquisition session and at the same time frame. The left eye appears normal with proper readings, but the right eye has lost its reading.

Moreover, we are experiencing a low success rate in eye calibration. The program keeps prompting us to change the nose pad or forehead pad, but it still fails even after trying all available pad combinations. Since we are only interested in pupil size and not gaze location, how important is it to complete the eye calibration? Additionally, do you have any suggestions for improving the calibration success rate?

Thank you very much!
Zipai

2

Interesting, can you please share how you are capturing the eye images?

To make sure you obtain an accurate pupil size reading, check the eye confidence value to make sure the cameras are able to detect the user’s eyes.

3

@ababilinski @wangzipai96 I can answer that as I am the dev that implemented the imagery capture for the project.

For some background:
The code used to capture the imagery was largely pulled from examples provided by the developer resources at the end of 2023.
At the end of 2024 we made the move to update the Unity project to use the ML SDK v2.5.0 along with OpenXR.

I don’t believe the aforementioned issue was encountered with earlier builds from before the migration, so I don’t know if perhaps something has changed with the API since then.
The image capture code in out project has not changed between the migration.

That said, here is the code that performs the imagery grab:

public override bool CaptureImagery()
    {
        if (!PermissionsManager.PermissionsGranted) return false;

        MLResult result = MLEyeCamera.GetLatestCameraData(out eyeCameraData, 0);

        // disregard when the cameras aren't showing all frames
        if (eyeCameraData.Frames.Length < MLEyeCamera.ActiveCamerasCount) return false;

        if (result.IsOk)
        {
            EyeCaptureFrameData eyeCaptureData = new() {
                frameData = new FrameData[eyeCameraData.FrameCount]
            };

            MLEyeCamera.EyeCameraFrameBuffer frameBuffer;
            FrameData newFrameData;

            for (int i = 0; i < eyeCameraData.FrameCount; i++)
            {
                frameBuffer = eyeCameraData.Frames[i].FrameBuffer;

                newFrameData = new()
                {
                    Width = (int)(frameBuffer.Stride / frameBuffer.BytesPerPixel),
                    Height = (int)frameBuffer.Height,
                    Size = frameBuffer.Size,
                    framePosition = eyeCameraData.Frames[i].CameraID,
                    Data = new byte[frameBuffer.Size]
                };

                MLEyeCamera.CopyImageFrameDataToByteArray(frameBuffer, ref newFrameData.Data);

                eyeCaptureData.frameData[i] = newFrameData;
            }

            frameData.Add(eyeCaptureData);
        }

        return true;
    }

The project utilizes the following:
Unity: 2022.3.42f1
Unity ML SDK: 2.5.0
ML OS: 1.10.0

4

When are you calling CaptureImagery()? Could you share a script that we can use to validate the issue?