Greetings,
we would like to be able to take a high resolution picture while recording video at a different resolution. Both should contain MR content. Is this possible with the current version of the SDK?
It seems like, if either the video or the picture should not contain MR content, this can already be achieved. Sadly, we need MR in both.
Unity Editor version: 2022.2.21f1
ML2 OS version: Release 2023 July
MLSDK version: 1.3.0
Host OS: (Windows/MacOS) Windows
Currently developers can only access the Mixed Reality content when using the Main Camera stream. I assume you are using the Magic Leap API to record content? If you only require 1 video stream and 1 photo stream you can create a stream configuration that captures Images and video at the same time.
var configs = new MLCamera.CaptureStreamConfig[2]
{
new MLCamera.CaptureStreamConfig()
{
OutputFormat = MLCamera.OutputFormat.JPEG,
CaptureType = MLCamera.CaptureType.Image,
Width = 1920,
Height = 1080
},
new MLCamera.CaptureStreamConfig()
{
OutputFormat = MLCamera.OutputFormat.YUV_420_888,
CaptureType = MLCamera.CaptureType.Video,
Width = 1920,
Height = 1080
}
};
MLCamera.CaptureConfig captureConfig = new MLCamera.CaptureConfig()
{
StreamConfigs = configs,
CaptureFrameRate = MLCamera.CaptureFrameRate._30FPS
};
I sadly am not able to use your proposed solution. It seems to be quite easy to mess up the configuration of the ML Camera. I would appreciate your help with getting the right setup to work.
We currently try to find a recording resolution that best matches 720p. I tried both your solution directly and to modify our existing one accordingly. You can see both code examples below.
This one did not work at all, with video only and with video and image set in the config:
MLCamera.ConnectContext connectContext = MLCamera.ConnectContext.Create();
connectContext.CamId = _identifier;
//MLCamera.Identifier.Main is the only camera that can access the virtual and mixed reality flags
connectContext.Flags = MLCamera.ConnectFlag.MR;
connectContext.EnableVideoStabilization = true; // Also doesn't work with pic+video if false
// For MR
connectContext.MixedRealityConnectInfo = MLCamera.MRConnectInfo.Create();
connectContext.MixedRealityConnectInfo.MRQuality = MLCameraBase.MRQuality._960x720;
connectContext.MixedRealityConnectInfo.MRBlendType = MLCamera.MRBlendType.Additive;
connectContext.MixedRealityConnectInfo.FrameRate = MLCameraBase.CaptureFrameRate._30FPS;
_camera = MLCamera.CreateAndConnect(connectContext);
// ...
//Initialize a new capture config.
var configs = new MLCamera.CaptureStreamConfig[2]
{
new MLCamera.CaptureStreamConfig()
{
OutputFormat = MLCamera.OutputFormat.JPEG,
CaptureType = MLCamera.CaptureType.Image,
Width = 1920,
Height = 1080
},
new MLCamera.CaptureStreamConfig()
{
OutputFormat = MLCamera.OutputFormat.YUV_420_888,
CaptureType = MLCamera.CaptureType.Video,
Width = 1920,
Height = 1080
}
};
// ...
_captureConfig = new MLCamera.CaptureConfig()
{
StreamConfigs = configs,
CaptureFrameRate = MLCamera.CaptureFrameRate._30FPS
};
MLResult result = _camera.PrepareCapture(_captureConfig, out MLCamera.Metadata metaData);
With that code, we always get MLResult_InvalidParam as a return value, even if we only pass a config for image or video respectively.
In the second attempt, which is close to our actual setup, we try to get the best fit for our desired resolution (720p) instead. The code is not that beautiful / a bit redundant. It works if only used for image / video, but not for both:
MLCamera.StreamCapability[] streamCapabilities = MLCamera.GetImageStreamCapabilitiesForCamera(_camera, MLCameraBase.CaptureType.Video);
MLCamera.StreamCapability[] streamCapabilitiesPhoto = MLCamera.GetImageStreamCapabilitiesForCamera(_camera, MLCameraBase.CaptureType.Image);
if (streamCapabilities.Length == 0 || streamCapabilitiesPhoto.Length == 0)
return;
//Set the default capability stream
MLCamera.StreamCapability defaultCapability = streamCapabilities[0];
MLCamera.StreamCapability defaultCapabilityPhoto = streamCapabilitiesPhoto[0];
//Try to get the stream that most closely matches the target width and height
if (MLCamera.TryGetBestFitStreamCapabilityFromCollection(streamCapabilities, captureWidth, captureHeight,
MLCameraBase.CaptureType.Video, out MLCamera.StreamCapability selectedCapability))
{
defaultCapability = selectedCapability;
}
if (MLCamera.TryGetBestFitStreamCapabilityFromCollection(streamCapabilitiesPhoto, captureWidth, captureHeight,
MLCameraBase.CaptureType.Image, out MLCamera.StreamCapability selectedCapabilityPhoto))
{
defaultCapabilityPhoto = selectedCapabilityPhoto;
}
//Initialize a new capture config.
var configs = new MLCamera.CaptureStreamConfig[2]
{
new MLCamera.CaptureStreamConfig()
{
OutputFormat = MLCamera.OutputFormat.RGBA_8888,
CaptureType = MLCamera.CaptureType.Video,
Width = defaultCapability.Width,
Height = defaultCapability.Height
},
new MLCamera.CaptureStreamConfig()
{
OutputFormat = MLCamera.OutputFormat.JPEG,
CaptureType = MLCamera.CaptureType.Image,
Width = defaultCapabilityPhoto.Width,
Height = defaultCapabilityPhoto.Height
}
};
_captureConfig = new MLCamera.CaptureConfig()
{
StreamConfigs = configs,
CaptureFrameRate = MLCamera.CaptureFrameRate._30FPS
};
We are thus still not sure how to enable both video and photo capture. Currently, we just use a frame of the video stream instead, but ideally we would like to take a higher resolution picture.
Thank you for the detailed explanation. Let me take a deeper look into this
@crucial-sailor I just got more information about this. Currently, we only support a single MR stream but would love you learn more about your use case and requirements for multi Mixed Reality Capture Streams so I can submit it to our voice of customer team.
If Mixed Reality Content is not required on the Camera Image, you can create an additional configuration where the mode is set to Camera Only and and the Camera ID is set to CV Camera. We will update our documentation to clarify this limitation.
Please let me know if you have additional questions
Thank you very much, then our use case is currently not possible - we require MR content both in the photo and in the video stream.
Simultaneous capture is relevant for us because we can take "screenshots" in our application in parallel to a running video stream. Depending on the platform, we resolve this by taking a single frame from the video stream, but whenever that's possible we prefer to take a photo with higher resolution and visual quality than we would get with a single frame from a video recording.
Pausing and resuming the video stream is not really an option, as that would take too much time.