Streaming of Mixed Reality Camera Frames with Timestamps

Unity Development
, , ,
1

Give us as much detail as possible regarding the issue you're experiencing:

Unity Editor version: 2022.3.42f1
ML2 OS version: 1.10.0
Unity SDK version: 2.5.0
Host OS: Windows

Hi,

I've been working with the MLCamera API for capturing mixed reality camera frames. Through my previous round of conversation with you (link to the old post), I was able to capture 648*720 MR frames at 30FPS and save it locally to the app dir. However, with some other functions implemented in my app, the FPS isn't very stable (e.g., 1 frame might get lost in ~2s), and I cannot really increase the resolution or frame rate since I could not save it fast enough with workers few enough to not slow down my app. I also tried to send the image frames via TCP to a python server (in the same WiFi network) but it even slowed down my app further.

Yet when I use the mixed reality streaming function in the ML Hub 3, it was very smooth (with very few corrupted frames). May I ask if you can share your implementation of streaming that in your ML Hub, or is there anything I can do to improve my image (video) capture logic? I do need the timestamp of the images since I need to align them with my eye tracking data. I'll attach the camera and the saving/sending code below for your reference.

MLCameraCapture.cs (18.1 KB)

AsyncImageSender.cs (7.3 KB)
AsyncImageSaver.cs (3.5 KB)

It would be very helpful if you could kindly advise on this matter. Thanks in advance!

2

Thank you for your feedback. We do not provide a sample for this type of encoding and sending. However, the issue you are most likely running into is with the performance limitation of saving each frame to a jpg or png before sending it over the network. A more efficient option would be to use Android APIs to encode and send the data as an H264 stream.

You can also take a look at Unity's WebRTC package and see if the package could be a solution to your issue:

3

Thanks for your suggestion! I'll take a look at this package