MLCamera.OutputFormat performance implications

WebRTC
1

Unity Editor version: 2022.3.1f1
ML2 OS version: 1.3.0
MLSDK version: 1.3.0

When creating a MLWebRTC.MediaStream from MLCamera, one can choose between YUV_420_888 and RGBA_8888 for output format. What are the performance implications of the choice?

Generally webcam devices produce YUV "natively", and the H.264 encoder requires YUV input. So typically the captured YUV data can be delivered to the encoder without transformation. But if I create a video source from a stream config specifying RGBA as the format, does that introduce per-frame transformations?

I would like to subscribe to MLCamera.OnRawVideoFrameAvailable to access local capture frames. We generally need to do this only momentarily to get a still photo with MRC (because my understanding is that the CV camera cannot do MRC). My hope is that everything is all YUV under the covers, with conversions to RGBA only as needed to supply subscribers of OnRawVideoFrameAvailable.

But of course I'm just guessing. Anything you can tell me about how this actually works will be greatly appreciated.

2

Hi @ks-scott,

I've reached out to our team regarding the cost of using either output format and I will report back to you as soon as I hear back.

Best,

El

3

As a rule of thumb, on ML2, if a given camera supports YUV_420_888 then it will should always perform better with the rest of the pipeline.

  1. If you are using Main or CV cameras, then you should be able to configure YUV_420_888 and feed the frames directly to ML2 video encoder without color conversion. If however you are using Mixed Reality camera, then it only supports RGBA_8888 through MLCamera. In this case, if you feed such a frame to video encoder, then the encoder will do color conversion before the compression takes place.
  2. If you are only interested in one Mixed Reality frame then the Mixed Reality camera can also output Still image buffer in jpeg format.
  3. If you are interested in getting a mixed reality frame via OnRawVideoFrameAvailable then you can either get RGBA_8888 or JPEG buffer.
4

Thanks for the quick reply. This is exactly what I was looking for, and is consistent with what we have observed.

1 Like