I have a Monobehavior that controls the MLCamera to capture frames in an app. I have been using the CV camera stream so that the Main camera stream remains free for use with Device Stream in the ML Hub. In the past, I have been using MLCamera.CaptureType.Video with the OnRawVideoFrameAvailable event handler to capture desired frames. I was looking to change my script to take advantage of the MLCamera.CaptureType.Image and the OnRawImageAvailable event so that I could use the CaptureImage or CaptureImageAsync method to request frames on demand, in hopes of getting a more stable image and if possible make use of the MLCameraBase.Metadata.SetControlAFMode() method to control the focus of the camera prior to requesting images, but I am running into some trouble.
I had planned on using MLCameraBase.OutputFormat.JPEG, but when I call PrepareCapture with the captureConfig that I've created, the MLResult returned is not Ok and gives the unhelpful InvalidParam code. I did a lot of testing, toggling lots of different properties and arguments trying to figure out which param of mine is invalid. I was trying to recreate the parameters in the Camera Capture example, but that is very difficult to disseminate based on the tight coupling with the UI in that example. That example allows you to use MLCamera.CaptureType.Image with MLCameraBase.OutputFormat.JPEG and the same resolution I was using. It took a while for me to realize that the only difference between the values I was setting in my code and those in the example was that I had switched to using the MLCamera.Identifier.CV stream a long time ago and the example uses the main camera. The CV Camera example only uses a video stream and not an image. As far as I can tell, there is not an example out there that uses both MLCamera.CaptureType.Image and MLCamera.Identifier.CV. But it should work. When I call GetStreamCapabilities on a camera that has been created using MLCamera.Identifier.CV, the list includes both IMAGE and VIDEO capabilities in a variety of resolutions. Finally, after enough poking around, I found by chance that when calling PrepareCapture after setting the output format to YUV_420_888, the result was Ok. Unfortunately, I am not interested in using YUV_420_888. It would require converting the color space on every image I request before moving on to my next step, and I imagine that is pretty processor intensive and could probably be handled better at a lower level where the GPU can be utilized to do that job.
Can you shed some light on what are the supported parameters and configurations for using MLCamera.CaptureType.Image with the CV camera stream?
Thanks!
Unity Editor version: 2022.2.5f1
ML2 OS version: 1.4.0-dev2
MLSDK version: 1.4.0-dev2
Host OS: (Windows/MacOS) MacOS
Error messages from logs (syntax-highlighting is supported via Markdown):