# What coud we know about frame camtoworld transformation accuracy, reliability and FrameTimestamps? **URL:** \ **Category:** General\ **Tags:** sensors, data-collection\ **Created:** [March 4, 2024, 2:11pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249 "2024-03-04T14:11:21Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![kocsisa](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@kocsisa](https://forum.magicleap.cloud/u/kocsisa)\ **Post date:** [March 4, 2024, 2:11pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/1 "2024-03-04T14:11:21Z")
For each frame data there is a "camToWorld transformation" / "pose" or in case of CV camera it can be queried by using an API call. **In optimal case the pose is calculated for the same moment when the frame was aquired.** **In practice how close it is to the optimal case?** Could we know how the pose is calcualted or estimated? E.g. it is determined in every 1/x sec, and each frame will get the closest one, or an estimation is calculated somehow? Are there timestamps used for some estimation calcualtion? Can we access them? What type of syncronization is used for this? How the speed or acceleration affects the accuracy and reliability? **About the frame timestamps** The Frame "MLTime FrameTimestamps" of each sensor/camera seems to be independent from eachother. They starting point are not the same and the frequency is also different. How could we get common timestamps which would mean the same time on the sam bases/epoch for all sensor/camera we use? Even the MLTime.ConvertMLTimeToSystemTime result does not seem to be such common timestamp. ---
**Author:** ![etucker](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/etucker/32/842_2.png) [@etucker](https://forum.magicleap.cloud/u/etucker)\ **Post date:** [March 4, 2024, 11:48pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/2 "2024-03-04T23:48:14Z")
> [@kocsisa](#): > > Are there timestamps used for some estimation calcualtion? Could you clarify what you mean by this? > [@kocsisa](#): > > **About the frame timestamps** > The Frame "MLTime FrameTimestamps" of each sensor/camera seems to be independent from eachother. They starting point are not the same and the frequency is also different. > How could we get common timestamps which would mean the same time on the sam bases/epoch for all sensor/camera we use? For synchronizing cameras, we recommend checking out [this solution](https://forum.magicleap.cloud/t/synchronous-capture-or-world-and-rgb-cameras-on-ml2/2149/5). ---
**Author:** ![kocsisa](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@kocsisa](https://forum.magicleap.cloud/u/kocsisa)\ **Post date:** [March 5, 2024, 12:30pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/3 "2024-03-05T12:30:02Z")
@etucker : For the second point: I tried to use the ConvertMLTimeToSystemTime method, but for me it seems, that it does not return a common "system" time. Let's see result from a recordings: ``` TimeStampSys ElapsedTSSeconds FrameType FrameNumber 191901473814 191,901473814 DepthShortRange 169 400564877422 200,282438711 WorldNormalRight 468 191865591241 191,865591241 CvVideo 205 400564877950 200,282438975 WorldNormalLeft 468 400564877810 200,282438905 WorldNormalCente 468 191941472628 191,941472628 DepthShortRange 170 191932267410 191,93226741 CvVideo 206 191981473928 191,981473928 DepthShortRange 171 191998928693 191,998928693 CvVideo 207 192021473486 192,021473486 DepthShortRange 172 400684877816 200,342438908 WorldNormalCente 471 400684877891 200,3424389455 WorldNormalRight 471 400684877825 200,3424389125 WorldNormalLeft 471 192061472488 192,061472488 DepthShortRange 173 192065604554 192,065604554 CvVideo 208 192101473610 192,10147361 DepthShortRange 174 400764876805 200,3824384025 WorldNormalLeft 473 400764879484 200,382439742 WorldNormalRight 473 400764876841 200,3824384205 WorldNormalCente 473 192141473692 192,141473692 DepthShortRange 175 192132263060 192,13226306 CvVideo 209 192181472653 192,181472653 DepthShortRange 176 192198930657 192,198930657 CvVideo 210 400884877325 200,4424386625 WorldNormalLeft 476 ``` The TimeStampSys values are calcualted by calling ConvertMLTimeToSystemTime .Investigating an hour long recordings, from the timestamp differences belonging to same framtypes we see that worldcamera systemtime tick is half-nanosec, the other cameras systemtime tick is 1 nanosec. Using this rule, I calcualted the second column:ElapsedTSSeconds. From this we can see that there are 8-9 seconds differences between different type of framtimestamps which follows each-other. On the sceen we recorded , there was a stopper; by this we checked that the recordings does not contain time-shifts, so the differences are coming from the timestamps only. Based on this we guess that the starting point (0 nanosec) for each sensor can be different. **So the question is:** After having the TimeStampSys values for each frames, how can we get for one frame all the other type of frames which are the most closer in real-time. Can we calcualte the real-time differences between timestamps if the frametypes are different? For your 1st question: e.g. in case of CV camera it is written, than poses are cached and frametimestamps is used for geting the pose. But how? It coud be the key in the cache, or somehow this timestamp is used for geting a global time which is used for selecting one "nearest in time" pose or calculating one pose from the previous and next in the time poses, by using speed vector, acceleration etc. ---
**Author:** ![etucker](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/etucker/32/842_2.png) [@etucker](https://forum.magicleap.cloud/u/etucker)\ **Post date:** [March 5, 2024, 9:07pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/4 "2024-03-05T21:07:51Z")
Could you perhaps provide more details on your use case? Are you trying to synchronize all of the data from the different cameras? This will help us get you towards a practical solution. If you are not comfortable sharing here, feel free to reach out to us directly. ---
**Author:** ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)\ **Post date:** [March 7, 2024, 4:28pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/5 "2024-03-07T16:28:17Z")
What frequency are you capturing the CV video at? You are correct, the pixel streams are not synchronized. Do you mind sharing information about your implementation ? All sensor timestamps should be coming from the same clock ---
**Author:** ![kocsisa](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@kocsisa](https://forum.magicleap.cloud/u/kocsisa)\ **Post date:** [March 8, 2024, 7:50am UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/7 "2024-03-08T07:50:34Z")
Hi It is not a must to get data from cameras at the same moment (if you mean that on syncing them), but when processing them on a server we need to collect all the camera/sensor information which are closest to a given moment or to eachother. Moreover we need to process data coming from external cameras and sensors too. So first we need to know which frame from e.g. Depth camera is the closest-in-time to e.g a given CV camera frame. Next we need to convert those timestamps to e.g. UTC ticks to get the closest-in-time information from external sources. **Lets see a situation:** - We capture CV video frames in CamOnly mode with resolution:720p @15FPS but process only 10 FPS , capture DepthShortRange @5 FPS, and WorldNormalCenter @10 FPS - A callback functin is called by ML when a CV frame is ready for processing. - A GetDepthFrames task calls _ **MLDepthCamera.GetLatestDepthData** _ to get the last frame, put it to the DepthQ and make the thread sleep as musch as necessary to reach the required FPS. - A GetWorldFrames task calls _ **worldCamera.GetLatestWorldCameraData** _ to get the last frame, put it to the WorldQ and make the thread sleep as musch as necessary to reach the required FPS. - A CV Compressor Task reads the CVQ comresses the frame and sends it to the WriterQ - A Depth Compressor Task reads the DepthQ comresses the frame and sends it to the WriterQ - A World Compressor Task reads the WorldQ comresses the frame and sends it to the WriterQ - the writer Task reads the WriterQ ands sends frames data and metadata to the target. e.g to a server To the server the next timestamps are sent currently for each frame: - FrameData.Timestamp - SysTimeStamp: _MLTime.ConvertMLTimeToSystemTime(frameData.TimeStamp, out long sysTimeStamp);_ - UtcTicks (The moment when the CV callback function or timed World/Depth request received the FrameData.) Based on The SysTimeStamps we see (as I sent earlier in this case) it seems that the starting points of the clocks for each frametype is different. So from this we can not tell which DepthFrame systimestamp value is the closest to which CVVideo systimestamp in the "real-world-time". ---
**Author:** ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)\ **Post date:** [March 8, 2024, 4:56pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/8 "2024-03-08T16:56:23Z")
Thank you for that detailed information. I have reached out to the relevant team for guidance and will keep you posted as soon as I hear back. ---
**Author:** ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)\ **Post date:** [March 25, 2024, 5:48pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/9 "2024-03-25T17:48:56Z")
Still working on this, and I will report back as soon as I have more information. We are investigating if there is a possible issue with ConvertMLTimeToSystemTime assuming a unified time step size, which could be causing the offset that you are seeing. Thank you for your patience. ---
**Author:** ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)\ **Post date:** [April 8, 2024, 5:54pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/10 "2024-04-08T17:54:05Z")
Would it be possible to get a sample of the script / project you are using to test with? ---
**Author:** ![kocsisa](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@kocsisa](https://forum.magicleap.cloud/u/kocsisa)\ **Post date:** [April 10, 2024, 2:40pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/11 "2024-04-10T14:40:35Z")
Hi, I attached a minimized Unity app which works on the same way as we get those timestamps in our full app and reproduce the same differences. When opening in Unity, If you requested to enter safemode and see 'protobuf' errors, than remove the "mlcamtest\Library\PackageCache\com.e7.protobuf-unity@948ab23d09" directory and wait for successful recompilation. You may need to set the project target for Android e.g. when Magicleap Project setup tool window opens. The code in **\Assets\Vartid\Scripts\RecorderMenu.cs** I attach a screenshot too which show the results. It can be seen if we compare The half of the WorldSysTs to Depth SysTs than the differences betwee 8-9 seconds, and the WorldSysTs counts twice faster than DepthTs. The CV SysTs is close to Depth Ts but it would be good to know if it is really is the same at the same real time or just close to the depth value. ![last_timestamps_in_secods](https://us1.discourse-cdn.com/flex019/uploads/magicleap1/original/2X/6/6852117b080d264e82841f82f5951dbd7396dbaf.jpeg) [mlcamtest.zip](https://forum.magicleap.cloud/uploads/short-url/fX62zlWnaXnZw8UHhdfXevruXCF.zip) (253.8 KB) ---
**Author:** ![kocsisa](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@kocsisa](https://forum.magicleap.cloud/u/kocsisa)\ **Post date:** [April 10, 2024, 2:46pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/12 "2024-04-10T14:46:54Z")
You may need to edit the Packages/manifest.json and set path for "com.magicleap.unitysdk": before opening the Unity project, The protobuf error on opening the project also solved if the protobuf is removed from this file before opening the project. ---
**Author:** ![kocsisa](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@kocsisa](https://forum.magicleap.cloud/u/kocsisa)\ **Post date:** [May 7, 2024, 12:22pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/13 "2024-05-07T12:22:33Z")
Hi @ kbabilinski, Did you have time for checking this app? Have you got the same difference? BR, András ---
**Author:** ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)\ **Post date:** [May 9, 2024, 7:19pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/14 "2024-05-09T19:19:16Z")
@kocsisa I haven sent the app to our QA team and asked them if they were able to reproduce the issue. I will let you know more as soon as I hear back. That said, they have not been able to reproduce this using a C++ application. One thing that could unblock you is using the new Pixel Sensor API which will become available later this month, we will be introducing a new API to obtain sensor data (outside of the Magic Leap Camera). ---
**Author:** ![kocsisa](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@kocsisa](https://forum.magicleap.cloud/u/kocsisa)\ **Post date:** [June 10, 2024, 12:03pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/15 "2024-06-10T12:03:30Z")
Hi @kbabilinski When trying to get the streams for "Picture Center" pixel sensor, the result (streamCount) is 0. And the current documentation says: "RGB Camera on the front of the headset ( **Only supports querying Camera Pose** )" Could we get the RGB camera streams via Piexel API in the next release? ---
**Author:** ![kocsisa](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@kocsisa](https://forum.magicleap.cloud/u/kocsisa)\ **Post date:** [June 10, 2024, 2:14pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/16 "2024-06-10T14:14:18Z")
Hi @kbabilinski One more Pixel API RGB camera related question: will pixel API support the current SetControlAERegions and SetControlAEExposureCompensation functionality? ---
**Author:** ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)\ **Post date:** [June 10, 2024, 4:19pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/17 "2024-06-10T16:19:10Z")
We are working on adding the ability to query the RGB Camera frames using the Pixel Sensor API however `SetControlAERegions` and `SetControlAEExposureCompensation` functionality (along with AHardware buffers) will require using the Android NDK APIs . That said, exposure control will be available in general as it is for other sensors. ---
**Author:** ![kocsisa](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@kocsisa](https://forum.magicleap.cloud/u/kocsisa)\ **Post date:** [June 18, 2024, 12:02pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/18 "2024-06-18T12:02:23Z")
Hi, @kbabilinski I started to rewrite the depth camera part, but I could not find any timestamp information for the depth frame. (Neither frame number or id) ``` pixelSensorFeature.GetSensorData( sensorId, stream, out PixelSensorFrame frame, out PixelSensorMetaData[] metaData, Allocator.Temp, shouldFlipTexture: true ) ``` (The original problem was that we had timestamp differences when using ML API, that is why you suggested the new Pixel Sensor API) Where should I search for the timestam ( and frameid)? ---
**Author:** ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)\ **Post date:** [June 19, 2024, 10:48pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/19 "2024-06-19T22:48:21Z")
XrPixelSensorDataML has XrTime captureTime but let me check with the Unity SDK team to make sure we expose it in the Unity SDK. ---
**Author:** ![kocsisa](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@kocsisa](https://forum.magicleap.cloud/u/kocsisa)\ **Post date:** [July 1, 2024, 9:34am UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/20 "2024-07-01T09:34:51Z")
@kbabilinski Have you received some info about getting timestamp and framenumber from the output of ``` pixelSensorFeature.GetSensorData( sensorId, stream, out PixelSensorFrame frame, out PixelSensorMetaData[] metaData, Allocator.Temp, shouldFlipTexture: true ) ``` I did not find any method or member about it in: - ``` UnityEngine.XR.OpenXR.Features.MagicLeapSupport.MagicLeapPixelSensorFeature ``` - ``` PixelSensorFrame ``` - ``` PixelSensorMetaData ``` ---
**Author:** ![kbabilinski](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/kbabilinski/32/19_2.png) [@kbabilinski](https://forum.magicleap.cloud/u/kbabilinski)\ **Post date:** [July 1, 2024, 6:37pm UTC](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249/21 "2024-07-01T18:37:07Z")
The XR time should be exposed in the next release of the SDK scheduled for later this month. [Next page](https://forum.magicleap.cloud/t/what-coud-we-know-about-frame-camtoworld-transformation-accuracy-reliability-and-frametimestamps/3249.md?page=2)