ML2 Rebooting

General
1

I’ve filed a bug report for this (request 38426) issue but am posting here for visibility in case anyone else has experienced anything similar.

We are seeing multiple of our ML2 devices spontaneously rebooting. It has been observed on the latest OS 1.12.0 as well as 1.11.0 for sure. I’m currently rolling one of the devices back to 1.10.0 to see if it is occurring there as well.

Anecdotally it is happening some time around when the device is losing / regaining head pose tracking but it’s not a direct reliably reproducible trigger. It could also just be encouraging the issue to happen sooner.

Bug report logs extracted after the reboot reveal a kernel panic related to the amd gpu driver.

I have attached a copy of one of these logs.

Please advise!

kpanic_dmesg_logs.txt (238.9 KB)

2

Do you notice this issue when running a specific application or service? Have you found that the device need to loose and regain head pose a few times within a short period for it to occur more frequently - or is loosing and regaining head pose once typically enough? Are you using an application that is resource intensive ?

3

We primarily notice this when running our own app just due to the fact that we are testing our own app most of the time, however one of our engineers has observed the reboot behavior once when running the Magic Leap provided Spaces app (our app was not running in the background).

Generally it seems that head pose needs to be lost and regained multiple times. Once is never enough to trigger this behavior.

One more bit of info for you is that we typically set our battery setting to Maximum Performance and we disable the Standby modes. Other than that, we are using the ML2 devices with default settings.

Final clues: we observed this issue sporadically when testing in our primarily indoors office environment, but then a week ago we had a demo at a client in a semi-outdoor space where this rebooting issue was encountered multiple times on multiple devices through the day. Ambient temperature was in the 60s F.

1 Like
4

[Part of same team]

I got it to occur by covering the cameras for 5 seconds and then uncovering. Did this multiple times while logging memory usage through adb. It didn’t reboot until several minutes later when I was entering the info and was wearing the device idle. Available system memory stayed stable at about 3.5gb. The reboots seem linked to a kernel panic with gpu driver. My case was something about cpu #6 accessing memory it couldn’t or something to that effect. System temperatures averaged around 32-55. 60 max for some items using our app.

1 Like
5

Got two reboots today. Our app was running and idling on desk. I don’t remember if the second reboot was running our app.

Side note: I had crashes on my pc with the newest intel cpu when doing some ai tasks. It was related to one cpu core trying to access memory in another and the bios blocks that. A firmware update (when it came out) fixed that. I don’t know if it has any relation to this situation though.

panic log snippet:

<4>[ 2260.367474] audit: audit_lost=332222 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2260.367480] audit: rate limit exceeded
<4>[ 2261.370940] audit: audit_lost=332457 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2261.370958] audit: rate limit exceeded
<4>[ 2262.380239] audit: audit_lost=332694 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2262.380243] audit: rate limit exceeded
<4>[ 2263.388806] audit: audit_lost=332931 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2263.388811] audit: rate limit exceeded
<4>[ 2264.400957] audit: audit_lost=333168 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2264.400962] audit: rate limit exceeded
<4>[ 2265.409705] audit: audit_lost=333405 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2265.409709] audit: rate limit exceeded
<4>[ 2266.421424] audit: audit_lost=333642 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2266.421429] audit: rate limit exceeded
<4>[ 2267.430551] audit: audit_lost=333879 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2267.430554] audit: rate limit exceeded
<4>[ 2268.441467] audit: audit_lost=334116 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2268.441472] audit: rate limit exceeded
<4>[ 2269.450730] audit: audit_lost=334353 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2269.450733] audit: rate limit exceeded
<4>[ 2270.462211] audit: audit_lost=334590 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2270.462219] audit: rate limit exceeded
<4>[ 2271.463358] audit: audit_lost=334825 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2271.463361] audit: rate limit exceeded
<4>[ 2272.471891] audit: audit_lost=335062 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2272.471895] audit: rate limit exceeded
<4>[ 2273.483607] audit: audit_lost=335299 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2273.483611] audit: rate limit exceeded
<4>[ 2274.491732] audit: audit_lost=335536 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2274.491736] audit: rate limit exceeded
<7>[ 2274.574719] mlmux-power MXPM1111:00: num status msgs received in last 20000 ms: 20
<4>[ 2275.495608] audit: audit_lost=335771 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2275.495611] audit: rate limit exceeded
<4>[ 2276.505798] audit: audit_lost=336008 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2276.505837] audit: rate limit exceeded
<4>[ 2277.507896] audit: audit_lost=336243 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2277.507899] audit: rate limit exceeded
<4>[ 2278.514763] audit: audit_lost=336480 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2278.514766] audit: rate limit exceeded
<4>[ 2279.528467] audit: audit_lost=336717 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2279.528471] audit: rate limit exceeded
<4>[ 2280.537660] audit: audit_lost=336954 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2280.537664] audit: rate limit exceeded
<4>[ 2281.539170] audit: audit_lost=337189 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2281.539173] audit: rate limit exceeded
<4>[ 2282.550565] audit: audit_lost=337426 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2282.550569] audit: rate limit exceeded
<4>[ 2283.552536] audit: audit_lost=337657 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2283.552539] audit: rate limit exceeded
<4>[ 2284.560909] audit: audit_lost=337894 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2284.560914] audit: rate limit exceeded
<4>[ 2285.564870] audit: audit_lost=338129 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2285.564874] audit: rate limit exceeded
<12>[ 2286.554195] healthd: battery l=100 v=8785 t=31.9 h=2 st=5 c=0 fc=5409000 cc=33 chg=a
<4>[ 2286.575359] audit: audit_lost=338366 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2286.575363] audit: rate limit exceeded
<4>[ 2287.582112] audit: audit_lost=338603 audit_rate_limit=5 audit_backlog_limit=64
<6>[ 5401.522532] mero_notifier: PID(3368) signal_cross_core: xpc_notification_send failed, xpc_target_mask=0x100, errno=-11
<6>[ 5401.534550] mero_notifier: PID(3368) signal_cross_core: xpc_notification_send failed, xpc_target_mask=0x100, errno=-11
<6>[ 5401.561594] mero_notifier: PID(3368) signal_cross_core: xpc_notification_send failed, xpc_target_mask=0x100, errno=-11
<6>[ 5401.599294] mero_notifier: PID(6038) signal_cross_core: xpc_notification_send failed, xpc_target_mask=0x100, errno=-11
<4>[ 5401.735620] RIP: 0010:0xffffb5b60020ccd0
<4>[ 5401.735627] Code: ff ff 00 b5 4f 33 ea ec ac 56 00 a8 01 e7 4d 8e ff ff fd ff ff ff 00 00 00 00 38 a9 01 e7 4d 8e ff ff 00 48 bf e0 4d 8e ff ff <01> 00 00 00 00 00 00 00 50 cd 20 00 b6 b5 ff ff 88 d3 29 c0 ff ff
<4>[ 5401.735632] RSP: 0018:ffffb5b6001e0ea8 EFLAGS: 00010282
<4>[ 5401.735639] RAX: ffff8e4dde4dad10 RBX: 0000000000000000 RCX: ffff8e4dde4daa10
<4>[ 5401.735647] RDX: 0000000000000101 RSI: 0000000000000000 RDI: ffff8e4dde4dad10
<4>[ 5401.735652] RBP: ffffb5b6001e0f50 R08: ffff8e4b086a09c0 R09: ffff8e4af2f77540
<4>[ 5401.735657] R10: ffff8e4af2f77240 R11: ffffb5b60020ccd0 R12: ffff8e4df317c1c0
<4>[ 5401.735664] R13: ffff8e4df56e9650 R14: ffff8e4df56e9600 R15: 000000000000000a
<4>[ 5401.735669] FS:  0000000000000000(0000) GS:ffff8e4df56c0000(0000) knlGS:0000000000000000
<4>[ 5401.735678] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[ 5401.735684] CR2: ffffb5b60020ccd0 CR3: 000000041cc16000 CR4: 0000000000340ee0
<0>[ 5401.735692] Kernel panic - not syncing: Fatal exception in interrupt
<0>[ 5401.736004] Kernel Offset: 0x2f400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
<3>[ 2287.582115] audit: rate limit exceeded
<4>[ 2288.585999] audit: audit_lost=338838 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2288.586003] audit: rate limit exceeded
<4>[ 2289.588681] audit: audit_lost=339073 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2289.588684] audit: rate limit exceeded
<4>[ 2290.601208] audit: audit_lost=339310 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2290.601213] audit: rate limit exceeded
<4>[ 2291.604984] audit: audit_lost=339545 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2291.604989] audit: rate limit exceeded
<4>[ 2292.609621] audit: audit_lost=339782 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2292.609625] audit: rate limit exceeded
<4>[ 2293.622113] audit: audit_lost=340019 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2293.622116] audit: rate limit exceeded
<4>[ 2294.623134] audit: audit_lost=340254 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2294.623140] audit: rate limit exceeded
<7>[ 2295.054738] mlmux-power MXPM1111:00: num status msgs received in last 20000 ms: 21
<4>[ 2295.632153] audit: audit_lost=340491 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2295.632158] audit: rate limit exceeded
<4>[ 2296.643418] audit: audit_lost=340728 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2296.643422] audit: rate limit exceeded
<4>[ 2297.650884] audit: audit_lost=340965 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2297.650887] audit: rate limit exceeded
<4>[ 2298.664680] audit: audit_lost=341202 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2298.664696] audit: rate limit exceeded
<4>[ 2299.672377] audit: audit_lost=341439 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2299.672382] audit: rate limit exceeded
<4>[ 2300.684332] audit: audit_lost=341676 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2300.684336] audit: rate limit exceeded
<4>[ 2301.693046] audit: audit_lost=341913 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2301.693049] audit: rate limit exceeded
<4>[ 2302.695058] audit: audit_lost=342148 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2302.695061] audit: rate limit exceeded
<4>[ 2303.706877] audit: audit_lost=342385 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2303.706880] audit: rate limit exceeded
<4>[ 2304.714457] audit: audit_lost=342622 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2304.714460] audit: rate limit exceeded
<4>[ 2305.715576] audit: audit_lost=342857 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2305.715580] audit: rate limit exceeded
<4>[ 2306.719340] audit: audit_lost=343092 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2306.719342] audit: rate limit exceeded
<4>[ 2307.731312] audit: audit_lost=343329 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2307.731316] audit: rate limit exceeded
<4>[ 2308.740113] audit: audit_lost=343566 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2308.740116] audit: rate limit exceeded
<4>[ 2309.751518] audit: audit_lost=343803 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2309.751521] audit: rate limit exceeded
<4>[ 2310.759803] audit: audit_lost=344040 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2310.759806] audit: rate limit exceeded
<4>[ 2311.772454] audit: audit_lost=344277 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2311.772457] audit: rate limit exceeded
<4>[ 2312.779727] audit: audit_lost=344514 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2312.779731] audit: rate limit exceeded
<4>[ 2313.792859] audit: audit_lost=344751 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2313.792863] audit: rate limit exceeded
<4>[ 2314.801646] audit: audit_lost=344988 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2314.801649] audit: rate limit exceeded
<7>[ 2315.534764] mlmux-power MXPM1111:00: num status msgs received in last 20000 ms: 20
<4>[ 2315.813083] audit: audit_lost=345221 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2315.813088] audit: rate limit exceeded
<4>[ 2316.820692] audit: audit_lost=345458 audit_rate_limit=5 audit_backlog_limit=64
<3>[ 2316.820695] audit: rate limit exceeded
<4>[ 2317.833592] audit: audit_lost=345695 audit_rate_limit=5 audit_backlog_limit=64```
6

I have the same reboot problems for several months, could not found the reason yet but the problem is lowered after disabling the multi-threaded rendering of Unity. I think reboot happens more frequent in early times of the glasses when it is turned on, after long use, reboot gets less. May be when ML2 is colder reboot happens more. I am almost sure problem is related with GPU.

7

Still looking into this issue for you. When updating the device to 1.12.0 did you go through the update path mentioned on the customer care page:
https://www.magicleap.care/hc/en-us/articles/10731234815757-Magic-Leap-OS-Update-Path

If you feel like the path was not followed, I would recommend grabbing one of the devices and trying to reinstall the OS version that the devices came with with factory reset. Then incrementing the OS version following the ML OS Update Path instructions until the headset reached OS version 1.12.0.

Assuming that the base OS was 1.0 the update path would be:

1.0.0 → 1.1.0 → 1.2.0 → 1.5.0 → 1.9.0 → 1.12.0

See the table for more details.

8

I sent you the serial numbers in a separate email.

Is there any harm in downgrading all the way back to 1.0.0 to update again through this recommended path? I don’t remember for sure which OS version my headset came preloaded with. I just don’t want to brick the device.

1 Like
9

Sorry for making you write the information twice, I missed the the ticket number that you linked in your initial post.

Just followed up with customer care and got access to your ticket ticket. You’re in good hands with Jack from customer care. I am going to work on it with him from their side. I forwarded your question regarding the upgrade path to them so they can help you through the steps.

10

@om-iman did reseting the device fixed your problem?

My unity app suffers from the exact same thing.
Disabling multi-threaded rendering helped, along with many other performance improvements, like render scale etc …But it still happens.

My app is not that heavy, its just a normal MRKT app with +/- 10 screens, and some components.

I dont think that this only happens with my app, and i found out this problem many months ago, but still havent found a way to completly fix it (if there’s any way …).

Its a really nasty behaviour. Even if the app were resource-intensive, the device should not reboot itself.

11

Have you tried updating your device to the latest version with the proper upgrade path?

12

I’m on the latest update, but i dont remember if i followed the proper upgrade path.

I was trying to figure out if resetting the device and following the upgrade path fixed the issue for others, before i reseted my device. But i’ll do it anyway and i’ll come back here once its done.

Thank you

13

Hi, we currently have 2 of our headsets which are randomly rebooting (and one also has lost it’s sound as well) and this is just showing the system menu - not even running an app. We have a support call with ML on this but I am now wondering if it might be related to OS updates. Whilst they have been updated to 1.12 with factory reset I do not know if they went through all the updates on the upgrade path. I had assumed that the upgrade path was only important if you wanted to keep settings and not factory reset but is that the case?