Unity Editor version: 2022.3.21f1
ML2 OS version: 1.5.0
Unity SDK version: 2.0.0
Host OS: (Windows/MacOS) Windows 11
With the newest SDK (2.0.0) release, we have big issues using MLCamera in Unity. We know it is deprecated, and are still hoping for new sample code to use the camera in Unity.
We use the MLCamera in combination with WebRTC (and MRTK 3). That has worked fine with Unity SDK 1.12.0, and it continues to work fine on OS 1.5.0 with this SDK.
As soon as we upgrade to SDK 2.0.0 (and also install com.unity.xr.magicleap 7.0.0), our camera class that is based on MLCamera will crash the app.
This error also happens without using WebRTC, if we use the sample code in the docs and Unity sample project to render the camera image directly on a MeshRenderer in Unity in our app.
-
If we render the video on the MeshRenderer, the app may crash at any time after the camera recording is started. Also, the app performance is going down severly, and the device framerate is affected significantly.
-
If we keep the code the same and only comment out the content in OnRawVideoFrameAvailable, thus not rendering the video, the app does not crash, and the app performance does not decrease.
-> It seems like "blocking" the frame callback from MLCamera by rendering the frames or passing them to WebRTC can cause a frame buffer overflow, which in turn causes the described behaviour.
The crash happens in between right when we start the camera and one minute after that.
The only relevant logs with priority "Warn" or higher that we get before the crash are:
2024/02/19 09:56:35.498 28149 28149 Warn UnityGfxDeviceW type=1400 audit(0.0:6986): avc: denied { search } for name="traces" dev="nvme0n1p37" ino=11534339 scontext=u:r:untrusted_app:s0:c110,c256,c512,c768 tcontext=u:object_r:trace_data_file:s0 tclass=dir permissive=0
Error messages from logs (syntax-highlighting is supported via Markdown):
2024/02/19 09:57:22.429 28149 9292 Error CRASH *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
2024/02/19 09:57:22.429 28149 9292 Error CRASH Version '2022.2.21f1 (4907324dc95b)', Build type 'Release', Scripting Backend 'il2cpp', CPU 'x86_64'
2024/02/19 09:57:22.429 28149 9292 Error CRASH Build fingerprint: 'ml/demophon_aosp/demophon:10/B3E.231116.09-R.096/100:user/release-keys'
2024/02/19 09:57:22.429 28149 9292 Error CRASH Revision: '0'
2024/02/19 09:57:22.429 28149 9292 Error CRASH ABI: 'x86_64'
2024/02/19 09:57:22.429 28149 9292 Error CRASH Timestamp: 2024-02-19 09:57:22.429905877+0000
2024/02/19 09:57:22.429 28149 9292 Error CRASH pid: 28149, tid: 9292, name: Binder:28149_1 >>> com.testapp <<<
2024/02/19 09:57:22.429 28149 9292 Error CRASH uid: 10110
2024/02/19 09:57:22.429 28149 9292 Error CRASH signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr --------
2024/02/19 09:57:22.429 28149 9292 Error CRASH rax 00007ea15f6e6870 rbx 00007ea15f6e6490 rcx 0000000000000000 rdx 0000100000000000
2024/02/19 09:57:22.429 28149 9292 Error CRASH r8 0000000000000b38 r9 00007ea15e956000 r10 00007ea2d97bbada r11 00007ea151d68b36
2024/02/19 09:57:22.429 28149 9292 Error CRASH r12 00007ea15f2f2d50 r13 0000000000000f30 r14 00007ea15f2f2d30 r15 00007ea15e95a370
2024/02/19 09:57:22.429 28149 9292 Error CRASH rdi 00007ea15f81fd48 rsi 00000000000000ec
2024/02/19 09:57:22.429 28149 9292 Error CRASH rbp 00000000000004c8 rsp 00007ea1c8300b40 rip 00007ea28539bb7e
2024/02/19 09:57:22.430 28149 9292 Error CRASH
2024/02/19 09:57:22.430 28149 9292 Error CRASH backtrace:
2024/02/19 09:57:22.430 28149 9292 Error CRASH #00 pc 000000000244fb7e /data/app/com.testapp-tvo_2NxnHJxRdIosH9rUfg==/lib/x86_64/libil2cpp.so (BuildId: ef9bab1bef2a2198)
2024/02/19 09:57:22.510 28149 28149 Warn Binder:28149_1 type=1400 audit(0.0:8077): avc: denied { getattr } for path="/dev/shregion" dev="tmpfs" ino=14500 scontext=u:r:untrusted_app:s0:c110,c256,c512,c768 tcontext=u:object_r:shregion_device:s0 tclass=chr_file permissive=0
2024/02/19 09:57:22.522 28149 28149 Warn Binder:28149_1 type=1400 audit(0.0:8079): avc: denied { getattr } for path="/dev/shregion" dev="tmpfs" ino=14500 scontext=u:r:untrusted_app:s0:c110,c256,c512,c768 tcontext=u:object_r:shregion_device:s0 tclass=chr_file permissive=0
2024/02/19 09:57:22.526 28149 28149 Warn Binder:28149_1 type=1400 audit(0.0:8081): avc: denied { getattr } for path="/dev/shregion" dev="tmpfs" ino=14500 scontext=u:r:untrusted_app:s0:c110,c256,c512,c768 tcontext=u:object_r:shregion_device:s0 tclass=chr_file permissive=0
2024/02/19 09:57:22.568 28149 28165 Error BufferQueueConsumer [BufferItemQueue] acquireBuffer: max acquired buffer count reached: 5 (max 4)
2024/02/19 09:57:22.568 28149 28165 Error BufferItemConsumer [BufferItemQueue] Error acquiring buffer: Unknown error -38 (-38)
2024/02/19 09:57:22.568 28149 28165 Error : acquireBuffer BufferItemConsumer::acquireBuffer fail, status -38
2024/02/19 09:57:22.573 28149 9292 Error CRASH Forwarding signal 11
0001/01/01 00:00:00.000 -1 -1 Info --------- beginning of crash
2024/02/19 09:57:22.573 28149 9292 Fatal libc Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x7ea15f6e6880 in tid 9292 (Binder:28149_1), pid 28149 (testapp)
2024/02/19 09:57:22.642 28149 28165 Error BufferQueueConsumer [BufferItemQueue] acquireBuffer: max acquired buffer count reached: 5 (max 4)
2024/02/19 09:57:22.642 28149 28165 Error BufferItemConsumer [BufferItemQueue] Error acquiring buffer: Unknown error -38 (-38)
2024/02/19 09:57:22.642 28149 28165 Error : acquireBuffer BufferItemConsumer::acquireBuffer fail, status -38
(The last 3 error messages are repeated a lot)
For now, this means that we need to keep using the old 1.12.0 Unity SDK. It would be great if the issues with the MLCamera could either be fixed, or if samples on how to get its frames in Unity with the C library, as well as compose virtual and real content and get extrinsics / intrinsics, are provided. As the MLCamera is getting deprecated, it would be great to have those samples.