Hi @etucker,
I did some more testing on our side and will try to format this response in two parts, one regarding Unity WebRTC + ML SDK, the other regarding MLWebRTC.
Unity WebRTC + ML SDK
Regarding the Google Drive Link
Thank you very much for offering to use the .aar built by you – we already have built and tested a version as well (for Unity WebRTC), and it should work:
We created a minimal OpenXR app without the ML SDK for the Magic Leap. There, we created a WebRTC peer and a local offer, and printed the resulting SDP. The whole process works fine.
Regarding Logs
We updated our Unity SDK to 1.8.0, and switched to version 1.3.0-dev2, just to use the latest software. The Magic Leap is still on the May 2023 version though.
As long as either the MLSDK or the .aar is part of the project, the app will not load. There is no Unity log message at all. Logcat does not give useful warnings or errors either.
Most likely while loading the app libraries, the app gets stuck (after 600ms) and does not even get far enough to start any Unity logging process.
The only logs that don’t seem meaningless are:
- 2023/07/19 09:45:21.241 6545 6545 Error […] Not starting debugger since process cannot load the jdwp agent.
- 2023/07/19 09:45:21.818 6545 6566 Error libprocessgroup set_timerslack_ns write failed: Operation not permitted
- 2023/07/19 09:45:21.879 6545 6566 Debug vulkan searching for layers in '/data/app/[…]==/lib/x86_64'
- 2023/07/19 09:45:21.879 6545 6566 Debug vulkan searching for layers in '/data/app/[…]==/base.apk!/lib/x86_64'
- 2023/07/19 09:45:21.898 6545 6545 Warn UnityMain type=1400 audit(0.0:259): avc: denied { search } for name="traces" dev="nvme0n1p37" ino=278531 scontext=u:r:untrusted_app:s0:c112,c256,c512,c768 tcontext=u:object_r:trace_data_file:s0 tclass=dir permissive=0 (shows 2x in a row)
- 2023/07/19 09:45:21.907 6545 6566 Info EGL-MAIN found extension DRI2_Blob version 1 (this is where the logs end)
As the app does not crash, we need to close it manually on the device. As said before, without either the .aar or the ML SDK the app actually starts, and shows our initial scene.
Did it work for you?
Can you observe a similar behavior in an app where both the Unity WebRTC SDK with the modified .aar for x86 and the ML SDK are present?
Regarding MLWebRTC
The WebRTC sample app
There is a WebRTC sample app, as documented in developer-docs.magicleap.cloud/docs/guides/unity/sdk-example-scenes/sdk-webrtc-scene#. I have a few issues to note about that app.
- For Python3, getargspec in bottle.py needs to be replace by getfullargspec, furthermore, while 127.0.0.1:8080 works, 0.0.0.0:8080/ does not work. That took a few minutes to fix, so I guess it would be helpful to write a hint about that in the developer guide.
- The app seems to work fine as long as the MagicLeap does not go into sleep mode during a call. This can already happen when the user looks elsewhere than through the glasses themselves. As soon as that happens, when the app wakes up from sleep again, the camera access is broken (if we’re lucky) or the whole system is broken (tracking is permanently lost on top of the broken camera, and we need to restart the device). Logs are shown below.
When the camera is “broken” that also means that the green recording indicator never disappears, so the camera access is not cleaned up properly, which also indicates potential memory leaks.
Can we disable the “sleep” behavior of the Magic Leap? It would be quite unfortunate if the app goes to sleep every time the user looks elsewhere than through the glasses. Of course, going to sleep when the device is not on somebody’s head makes sense, but, coming from the HoloLens, it would be quite nice if the device would stay awake no matter what the eye tracking system is measuring.
Using MLWebRTC in our app
Compared to native and Unity WebRTC, we noticed that some interfaces were slightly changed (e.g. for setting ICE servers during peer creation), while others are completely missing.
We don’t need all configurations, but the following issues remain:
- Codec support: We need to support more than just H264. Unity WebRTC, for example, supports VP8 as well. Software encoding would be sufficient here.
- As with the sample app, as soon as the app goes to sleep, we run into issues regarding the camera and microphone access.
- Destruction/teardown and the whole setup was not clear to us, due to lacking documentation. It took some trial and error to not leave the camera open when closing a WebRTC call, and we are still not sure if there are potential memory leaks depending on the order in which the camera and tracks are closed.
