Accessing MagicLeap 2 Camera and Microphone

Unity Development
,
1

Give us as much detail as possible regarding the issue you're experiencing:

Unity Editor version: 2022.3.19
ML2 OS version: 1.5
Unity SDK version: 1.12
Host OS: (Windows/MacOS) Windows 11

I tried using MLAudio and MLCamera to capture audio and video from the Magic Leap live but I can't seem to get the data. I want to take the audio live and store it in a .wav. And the video needs to be taken live and also stored in a .mp4 file. Can someone help me with this.

These are the links for the code I used:

2

Have you enabled and requested the permissions at runtime? Have you tested the Capture example in the Magic Leap Unity Examples v 1.12.0 project?

3

All permissions are enabled.

I tried testing the AudioCapture scene and it froze with this error message:
Error: MLAudioGetBufferedInputDefaults in the Magic Leap API failed. Reason: MLResult_NotImplemented
UnityEngine.XR.MagicLeap.MLAudioInput/StreamingClip:.ctor (UnityEngine.XR.MagicLeap.MLAudioInput/MicCaptureType,int,int)
MagicLeap.Examples.AudioCaptureExample:StartMicrophone () (at Assets/MagicLeap/Examples/Scripts/AudioCaptureExample.cs:214)
MagicLeap.Examples.AudioCaptureExample:HandleOnTriggerDown (UnityEngine.InputSystem.InputAction/CallbackContext) (at Assets/MagicLeap/Examples/Scripts/AudioCaptureExample.cs:329)
UnityEngine.InputSystem.LowLevel.NativeInputRuntime/<>c__DisplayClass7_0:<set_onUpdate>b__0 (UnityEngineInternal.Input.NativeInputUpdateType,UnityEngineInternal.Input.NativeInputEventBuffer*)
UnityEngineInternal.Input.NativeInputSystem:NotifyUpdate (UnityEngineInternal.Input.NativeInputUpdateType,intptr)

Error: MLAudioCreateInputFromMicCapture in the Magic Leap API failed. Reason: MLResult_NotImplemented
UnityEngine.XR.MagicLeap.MLAudioInput/StreamingClip:.ctor (UnityEngine.XR.MagicLeap.MLAudioInput/MicCaptureType,int,int)
MagicLeap.Examples.AudioCaptureExample:StartMicrophone () (at Assets/MagicLeap/Examples/Scripts/AudioCaptureExample.cs:214)
MagicLeap.Examples.AudioCaptureExample:HandleOnTriggerDown (UnityEngine.InputSystem.InputAction/CallbackContext) (at Assets/MagicLeap/Examples/Scripts/AudioCaptureExample.cs:329)
UnityEngine.InputSystem.LowLevel.NativeInputRuntime/<>c__DisplayClass7_0:<set_onUpdate>b__0 (UnityEngineInternal.Input.NativeInputUpdateType,UnityEngineInternal.Input.NativeInputEventBuffer*)
UnityEngineInternal.Input.NativeInputSystem:NotifyUpdate (UnityEngineInternal.Input.NativeInputUpdateType,intptr)

Error: MLAudioStartInput in the Magic Leap API failed. Reason: MLResult_NotImplemented
UnityEngine.XR.MagicLeap.MLAudioInput/StreamingClip:.ctor (UnityEngine.XR.MagicLeap.MLAudioInput/MicCaptureType,int,int)
MagicLeap.Examples.AudioCaptureExample:StartMicrophone () (at Assets/MagicLeap/Examples/Scripts/AudioCaptureExample.cs:214)
MagicLeap.Examples.AudioCaptureExample:HandleOnTriggerDown (UnityEngine.InputSystem.InputAction/CallbackContext) (at Assets/MagicLeap/Examples/Scripts/AudioCaptureExample.cs:329)
UnityEngine.InputSystem.LowLevel.NativeInputRuntime/<>c__DisplayClass7_0:<set_onUpdate>b__0 (UnityEngineInternal.Input.NativeInputUpdateType,UnityEngineInternal.Input.NativeInputEventBuffer*)
UnityEngineInternal.Input.NativeInputSystem:NotifyUpdate (UnityEngineInternal.Input.NativeInputUpdateType,intptr)

For the Camera Capture, same result with:
Error: MLMediaPlayerCreate in the Magic Leap API failed. Reason: MLResult_NotImplemented
UnityEngine.XR.MagicLeap.MLMedia/Player:.ctor (UnityEngine.XR.MagicLeap.MLResult&)
MagicLeap.Core.MLMediaPlayerBehavior:get_MediaPlayer () (at Library/PackageCache/com.magicleap.unitysdk@98db447e4af9/Runtime/Deprecated/MediaPlayer/Scripts/MLMediaPlayerBehavior.cs:84)
MagicLeap.Core.MLMediaPlayerBehavior:Reset () (at Library/PackageCache/com.magicleap.unitysdk@98db447e4af9/Runtime/Deprecated/MediaPlayer/Scripts/MLMediaPlayerBehavior.cs:385)
MagicLeap.Examples.CameraCaptureExample:OnApplicationPause (bool) (at Assets/MagicLeap/Examples/Scripts/CameraCapture/CameraCaptureExample.cs:179)

Exception: Media Player initialization result with error MLResult_NotImplemented.
MagicLeap.Core.MLMediaPlayerBehavior.get_MediaPlayer () (at Library/PackageCache/com.magicleap.unitysdk@98db447e4af9/Runtime/Deprecated/MediaPlayer/Scripts/MLMediaPlayerBehavior.cs:108)
MagicLeap.Core.MLMediaPlayerBehavior.Reset () (at Library/PackageCache/com.magicleap.unitysdk@98db447e4af9/Runtime/Deprecated/MediaPlayer/Scripts/MLMediaPlayerBehavior.cs:385)
MagicLeap.Examples.CameraCaptureExample.OnApplicationPause (System.Boolean isPaused) (at Assets/MagicLeap/Examples/Scripts/CameraCapture/CameraCaptureExample.cs:179)

ArgumentException: Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection.
System.Buffer.BlockCopy (System.Array src, System.Int32 srcOffset, System.Array dst, System.Int32 dstOffset, System.Int32 count) (at <8f06425e63004caf99a79845675f751e>:0)
UnityEngine.XR.MagicLeap.MLCameraBase.FlipFrameVertically (UnityEngine.XR.MagicLeap.MLCameraBase+CameraOutput& frame) (at Library/PackageCache/com.magicleap.unitysdk@98db447e4af9/Runtime/Deprecated/Camera/API/MLCameraBaseHelpers.cs:92)
MagicLeap.Examples.CameraCaptureVisualizer.OnCaptureDataReceived (UnityEngine.XR.MagicLeap.MLCameraBase+ResultExtras extras, UnityEngine.XR.MagicLeap.MLCameraBase+CameraOutput frameData) (at Assets/MagicLeap/Examples/Scripts/Visualizers/CameraCaptureVisualizer.cs:167)
MagicLeap.Examples.CameraCaptureExample.OnCaptureRawImageComplete (UnityEngine.XR.MagicLeap.MLCameraBase+CameraOutput capturedImage, UnityEngine.XR.MagicLeap.MLCameraBase+ResultExtras resultExtras, UnityEngine.XR.MagicLeap.MLCameraBase+Metadata metadataHandle) (at Assets/MagicLeap/Examples/Scripts/CameraCapture/CameraCaptureExample.cs:675)
UnityEngine.XR.MagicLeap.Native.MLThreadDispatch+DispatchPayload3`3[A,B,C].Dispatch () (at Library/PackageCache/com.magicleap.unitysdk@98db447e4af9/Runtime/Common/Utils/MLThreadDispatch.cs:529)
UnityEngine.XR.MagicLeap.Native.MLThreadDispatch.DispatchAll () (at Library/PackageCache/com.magicleap.unitysdk@98db447e4af9/Runtime/Common/Utils/MLThreadDispatch.cs:287)
UnityEngine.XR.MagicLeap.MLDevice.Update () (at Library/PackageCache/com.magicleap.unitysdk@98db447e4af9/Runtime/Common/Utils/MLDevice.cs:519)

I have not made any changes to the code at all. All project settings permissions are granted.

4

The API does not support Magic Leap application simulator. Please use the Unity Microphone class instead.

5

Is there a different API that does Camera and Audio recording (video recording) that supports Application Simulator?

6

Unfortunately we do not support that functionality but I will let our voice of customer team know. Do you mind sharing more information about your application?

7

The application will take in video and audio input and feed it into an AI model. The model is going to be run on a PC that also has the Unity scene running the application. That is why the application simulator is necessary as this application will not be able to be put in an .apk file

8

Remote Render might be a better alternative for your use case rather than App Simulator. That said, remote render does not support the Video or Audio streams at this time either so I will make sure to share your request with the voice of customer team.