After several minutes of capturing the CVCameraPose hangs the thread.
It runs on the main thread in a callback method, as in the example project.
After several minutes the screen is not refreshed (nothing can be seen), in the log we can see some error message (see below) before the hanging-up.
if I replace this
mlResult = MLCVCamera.GetFramePose(resultExtras.VCamTimestamp, out cameraTransform);
if (mlResult.IsOk)
with this
cameraTransform = new();
than the app works "properly" - certeanly with wrong pose data - , and captures the frame data for more than 30 minutes.
Is it the same problem, a new one, or I just need to upgrade something?
Unity Editor version: 2022.2.0b16.112.5806 ML2 OS version: 1.3.0-dev2 MLSDK version: 1.8.0 Host OS: Windows 11 Pro 22H2
Error messages from logs
Error: MLCVCameraGetFramePose in the Magic Leap API failed. Reason: MLResult_UnspecifiedFailure
Error: MLCamera.InternalGetFramePose failed to get camera frame pose. Reason: MLResult_UnspecifiedFailure
@kocsisa Were you able to reproduce this in the Unity Examples Project. I have not experienced this issue. Are you recording a video or capturing an image?
The MLCVCamera.GetFramePose() function can fail if more than 500ms passed since the image was captured. This can occur when the framerate drops below 60. If you think this is occurring even when the frame rate is stable, do you mind sending us a sample project we can test?
In the Unity Example Project, it might not be possible to reproduce, as the Example do not get all data
from all cameras Simultaneously. Currently I'm working on reducing the code and the recorded data to catch the edge when it comes up, than come back to you. Meanwhile I'm upgrading Unity to 2022.2.19f1 and maybe later to 2022.2.21f1. I will come back to you with the results.
Until that as a new info: just starting the app which has a primitive startup screen only and camera initialization as in the examples - with a start and stop buttons, so not a complicated one - without starting the recording, the profiler always shows around 60fps which has short spikes below 30 fps. It is almost the same when start recording and recording the data. I attach a profiler screenshot made at the point of freezing.
Hi @kocsisa regarding the sample scene you are profiling. Is this sample only activating the RGB camera or is it activating the World, RGB, Depth and eye cameras all at once ? It's also possible that the spikes are coming from the profiler's overhead.
@kocsisa thank you for this information . Do you mind sharing a debug version of your application with me? You can send a link to it as a private message as well.
Does this issue occur even when you are not visualizing the resulting texture? To improve performance you can also try querying the Depth and World camera images on a separate thread.
Next week -after my holiday- I will give it a try to capture CV cam only., if it works than I move the depth and world cam querying to sepsrate thread from the update. The captured data is not visualized, but saved on separate threads.
We could reproduce the same behaviour (with same or very very similare results) by adding similare save functionality to the Example app "CameraCapture" Scene , even to the latest version.
Before the tests the free space on the disk is more than 200 GB.
First we tested on Version 1.3.0-dev2 Build B3E.230526.01-R.026
Then we tesed on Version 1.3.1 Build B3E.230616.02-R.121 using the latest example code too and Unity Editor version : 2022.2.0f1
Start the modified Example:
start "CameraCapture" Scene
set parameters: CamOnly, Video, 30FPS, RGBA_888, 1280x720, (Do not record to file)
press "Capture (10s)" will start recording for 1000 seconds.
Test rounds:
After writing 34GB the main thread hangs (empty screen). The writing thread starving and write warning about it to Unity Debug.log. detaiiled log attached.
Restarting the app and recording results the same but after writing only ~1.5 GB
Restarting again; results the same but after writing only ~0.5 GB
Restarting again; results the same but after writing only ~0.75 GB
Move the files from /sdcard/Android/data/com.magicleap.unity.examples/files to /sdcard/Android/data/Tempnewfolder then rastarting the app; results the same but after writing only ~0.6 GB
Deleting files from /sdcard/Android/data/Tempnewfolder and leaving the small (~0.6 Gb) file in /sdcard/Android/data/com.magicleap.unity.examples/files; then rastarting the app; results the same but after writing ~28GB
Deleting all files from /sdcard/Android/data/com.magicleap.unity.examples/files; then rastarting the app; results the same but after writing ~34GB (the same amount as in the first test!)
Remove the added GetFramePose related lines form "CameraCaptureExample.cs: OnCaptureRawVideoFrameAvailable" (mlResult = MLCVCamera.GetFramePose(resultExtras.VCamTimestamp, out cameraTransform);if (mlResult.IsOk) ...);
then redeploy and start the app;
results that you can record the whole 1000 seconds (~94GB) even if we had the previously recorded large 34GB file in the app's dir.
The FPS drops to very low frequently, but the app not hangs forever just we have 1-2 sec long lags. But the app can continue running after the lag and continue saving frames.
the test result the same if we restart MagicLeap too.
the recordable amount is decreased when we upgraded to the latest 1.3.1 firmware and used the latest example project. (29 GB instead of 34 GB)
if we change the FileStream to behave as a devNull writer, than also everything works with calling GetPose too -(certeanly the output file will be empty in that case)
As I see logs show the same in all cases when the recording stops and the main thread hangs.
I Attachedd the CameRaCapture directory with an added and a modified file in the zip, and the logs in txt.
I hope you can tell something, what couse this strange behaviour, and we could step forward
András
I tested your application and can confirm the results however my application only records for a few seconds before crashing. I will continue to investigate but this is most likely due to the file stream using too many resources since I am able to record longer if the image size is smaller. Saving the raw data provided by our camera every frame is resources intensive, have you considered using an encoder or our media capture API to save the data into a video file?
What you wrote, seems to be the same as we recognized. i.e., If you decrease some parameters (and by this the Mb/s to be written to the disk) it increases the time you can record, but not the total amount of datan in GB. (if the starting env /e.g. disk space/ is the same)
For our goals we must have the raw data with lossless compression. And we have minimum FPS and resolution for this too, moreover we need to have almost all the data from all censors for each frame.
In our app we use compression (~40-80% depending on frame-type), but it only means that we could record some more minutes, but we need to record at least 30-90 minutes.
The file-writing itself should not over-helm the system, as it uses only one separated thread and we do not push a new frame into the Q if it reached the allowed size (50). So, we just skip the frames if there are 50 of them waiting for the writer.
And if we remove the GetPose, then the system can write out all the data, and not crashes the app.
And the more strange things are:
why we could record more whn we delete as many file as possible, meanwhile we would have plenty of free space without this too.
why we could write as many data as we want if we do not use the getPose?
So as I feel, not the file writing itself cause the problem, just help other problem to come to the surface.
@kocsisa Thank you for that information. I have sent your feedback and information about this issue to a additional internal team members. I will keep you updated as soon as I have more information
To make sure I am experiencing the same issue, I noticed that when you do not use the GetPose function you are able to continue recording however the app begins to hang intermittently. Dropping way below 60fps.
Hi Kocsisa, the bug has been reported and is being investigated. Is this bug blocking development. Are you able to reduce the frame rate while we work on resolving this?
@kocsisa note I cannot guarantee that this can be accomplished given the performance overhead of writing the raw data to the device at your target a resolution. If you record at a lower quality would you be able to continue development? Have you considered sending the bytes to a separate device and writing the data there to reduce overhead?
