Exclusive Rendering in Secondary View?

Unity Development
,
1

Unity Editor version: 6000.2.10f1
ML2 OS version: 1.12.0
Unity SDK version: 2.6.0
Host OS: Windows

Hi,

This is kind of a follow-up for this post made a while back. Our goal is still to capture where the eye gaze lands on the recorded video, which inevitably need two camera streams, one for capturing the video and the other for recording timestamps and camera transform and intrinsics. Last time we resolved the issue by stopping Marker Tracking and then initialize the cameras, however for the current app we need to enable Marker Tracking at all times and therefore cannot utilize the old solution. And using world camera could not release the CV camera too. We have made three attempts to resolve this:

  1. We tried to, instead of capturing videos, capturing images and save it locally. However this has to work at an extremely low frequency to ensure the performance of the app, which is lower than the acceptable threshold.
  2. We wondered if the main camera and the ML Camera are a fixed offset from each other with a very similar rotation. This appears not to be the case.
  3. Following suggestions provided by AI, we figured that if we can simply render an object at gaze point and let the video stream record it, We can also know where it lands on. However, we don’t want this object to appear in the user’s view. AI suggested we put this object on a separate layer which the main camera’s culling mask does not include, however when I did that and set the object’s renderer to be enabled when capturing is on by fixationCube.GetComponent<Renderer>().enabled = MLXrSecondaryViewState.IsActive;, it simply does not show anything in the video stream either. I tried to use ML Hub 3 as well as our local code for video capture, but neither captures this “invisible object”.

Given my limited knowledge on ML Camera, could you please advise on how we can achieve our goal? It appears to me that the third option we tried is the most promising, however I’m not sure how to configure it such that the object is rendered exclusively in the “secondary view”. Is there any solution to this problem?

Thanks!

Best,

Zhehan