WebRTC Call Latency Issues on Magic Leap 2

Unity Development
,
1

Hello fellow developers,

I am currently working on a project involving the implementation of a WebRTC Call functionality using Magic Leap 2 SDK examples of WebRTC. However, I've encountered an issue where I am experiencing a significant delay in WebRTC calls, sometimes exceeding 3-5 seconds.

This is unexpected because, typically, WebRTC P2P calls only have a maximum latency of 20-50ms based on the TURN Service Provider. For reference, I am utilizing the Twilio NTS Service for NAT Traversal, which offers global latency.

Latencies of two different endpoints before WebRTC Communication:

image
image1474×272 16.4 KB

I've closely followed the Magic Leap 2 SDK examples of WebRTC, without making any modifications. Furthermore, I've noticed a similar type of delay when using Magic Leap's own examples.

This makes me believe there might be an underlying issue causing this substantial delay, although I am uncertain of the root cause at this point. To aid in the investigation, I am providing the latencies of my APIs and related system information:

  • Unity Editor version: 2022-2.0f1
  • ML2 OS version: Version 1.3.0-dev1, Build B3E.230427.10-R.023, Android API Level 29
  • MLSDK version: 1.7.0
  • Host OS: Windows 11

Moreover, if there are any specific error messages from the logs that could be helpful, I can provide them as needed.

Any guidance or suggestions from the community would be greatly appreciated. Thank you.

Best Regards,

Muhammad Usman Bashir
Lead AR Developer

2

Hi @usman.bashir, please include diagnostic logs and we'll take a look.

3

I have uploaded the logs file of my testing the WebRTC calling.

ML2-WebRTC-Logs.txt (8.9 MB)

4

@kdowney did you find time to look into the issue?

For your better understanding, this issue is coming in between the intervals. I am posting a video URL for your reference, where you can see the hand movement delay on local stream and remote stream:

5

Hi @usman.bashir It's in the queue. Thank you for your patience.

6

Thank you for providing video demonstration of this issue. If I wanted to minimally reproduce this issue, what steps should I take?

7

Simply run the MagicLeap_Examples related to WebRTC Sample Scene. Make a direct call between Device and Browser, you will observe the things.

8

Thank you for this. I will go ahead and diagnose this issue and let you know of my findings.

9

I see that you shared your local logs. Would it be possible to share the server logs as well?

10

@usman.bashir Some findings related to @etucker's request above for the server logs:

We spoke to our WebRTC team and analyzed the logs you sent. We do not see anything abnormal so it would help to take a look at the server logs.

We see:

  • The call was initiated at 18:00:39
  • Webrtc connection started at 18:00:40
  • Connection was estabilished at 18:01:08

The webrtc signaling process takes 28 seconds and it is normal considering the network and server latency. If it takes minutes then it would be a concern. And it is expected behavior.

Log snippets for this sequence.

`2023/06/22 18:00:39.366 6498 6525 Info Unity MagicLeap.Examples.MLWebRTCExample:Connect(String)
2023/06/22 18:00:39.366 6498 6525 Info Unity CallConnect:Start() (at D:\Android\AR\wavefunction\ConceptHealth-A-Consult\Assets\A-Consult\Assets\Scripts\UI\Forms\CallConnect.cs:14)

2023/06/22 18:00:40.958 6498 6525 Info luminkarrot nova/frameworks/kali/kaliwebrtc/simple/src/Factory.cpp:71 Creating peer connection
2023/06/22 18:01:08.017 6498 6681 Info luminkarrot nova/frameworks/kali/kaliwebrtc/simple/src/Connection.cpp:93 Peer Connection State: --> connected 2023/06/22 18:01:08.017 6498 6681 Info luminkarrot nova/frameworks/kali/kaliwebrtc/simple/src/Connection.cpp:356 Connection State: 4 (setting remote answer) --> 9 (connected)`

If we get server logs, we can say where that 28 seconds was taken.

Also, we have seen behavior like this in our testing when on our corporate network. We eliminated this by switching to a dedicated wifi network.

Please note that the signaling service that we use is not production quality and only given as a reference.

11

Thank you for the detailed insights and for taking the time to analyze the logs I provided. Your feedback helps me to better understand the processes at work during our Expert call tests.

Your analysis of the signaling process, and the information that 28 seconds is normal for connection establishment considering network and server latency, is particularly useful to me. I would like to share that our cloud machine is deployed in the UK region, and I was testing the calling feature from Pakistan, which would explain some latency. We have not yet optimized our cloud machine for global low latency, but it's definitely something we will consider in the future.

Moreover, I agree with your point about the challenges with WebRTC call intervals when video frames slow down. This is something we are also observing. During our tests, we found that approximately 75% of calls are completed without issue, but in the remaining 25% of calls, we have noticed delays. I am actively working to resolve this issue.

I am testing the current feature by changing resolution and fps to better observe this latency issue. Second thought came into my mind. Can I enable hardware acceleration codec for WebRTC call?

12

It is possible to enable hardware acceleration codec for WebRTC calls. Yes.

13

@etucker

I. Introduction:
A series of more than 10 comprehensive tests were conducted on a ML2 device.

II. Test Specifications:

Last two tests were long duration tests.

The final test was executed for a total duration of 1 hour.

III. Test Conditions:

  • The device was completely charged at the beginning of the test, showing a 100% battery status.
  • After the 30 minutes of test performed, Battery status was turned to 97%.
  • I put the application to background and open settings. Frame become inactive, then I again open Application using controller. Frame again resumed.

IV. User Interactions:

  • An interesting aspect of the test was that the scene on the device would pause if the user wasn't wearing glasses.
  • Once the device was picked up again by the user, the scene would reactivate and resume.

V. Test Observations:

  • Intermittent video latency * was experienced during the test.
  • These latency periods lasted between 10 to 20 seconds each.
  • After each latency period, the video frames would recover automatically without any user intervention.

VI. Conclusions:

The tests provided valuable insights into the device's performance under specific conditions, which will help in further refining its performance and user experience.

1 Like