We are working on a project using UE 5.4 (commercial release, Remote Rendering only) and ML2. We want to take cad models and Align them to their Large real world counter parts using UE blueprints. Due to sensitive data, we can not use anything cloud. Essentially user puts on headset look at real world object (18 foot long) and AR model pops overtop and stays aligned as user walks around, steps inside of physical mockup model inside hatches and inside panels...
What would you guys recommend, spatial anchoring, Aurco Markers, OpenCV or something else?
How large are the objects? You could use the marker tracker that's included in the ML OS. However, you may want to use more than one marker or combine marker detection with a manual refinement process to align your content if the objects are very large.
The ML UE sdk has a marker tracking plugin. We have a short guide on using it here-
This page has information on the general capabilities of the marker tracker-
If the physical environment where the objects are located is always the same, you could also try using the on device spatial mapping to create a map of the environment that the ML2 can localize into. You can use the Spaces app to create local maps up to ~250m^2. Mapping large spaces requires a connection to an AR Cloud server, which you could host on premises if you wanted to.
Note- remote rendering does not currently support spatial anchors, so the experience will be a bit basic. You can use the "Magic Leap Localization Map" plugin, which should give you a consistent world origin relative to the physical environment when the device localizes into the space, but you would have to determine and persist the location of objects relative to the origin yourself.
Marker tracking is probably easier and more appropriate if you need a way to track individual objects vs the devices location in an environment, especially if the objects can move.
Thanks for the Quick Reply. I think Aruco markers is def the way to start out. I plugged in the blueprints from your link and printed out the aruco markers. Should It just pop up a xyz cord gizmo on top of the aurco in AR? I am not able to get the ML2 to scan the Aruco print outs. Maybe I am missing a step or setting somewhere?
np. Hm, that looks like it should work. Yes, that should render a coordinate gizmo when a marker is detected. Did you add the marker tracker permission?
It's com.magicleap.permission.MARKER_TRACKING
You could also try setting the marker length in the configuration to the length of the edge of the markers (in cm) you are trying to detect.
Actually, I'm sorry. I did some testing here and it looks like there are some issues-
First, sorry I forgot that you were only using remote rendering. You don't need to add the MARKER_TRACKING permission. It won't make any difference since you're actually targeting Windows.
Second, after adding the marker tracker component, I believe you need to Activate it. We'll update the docs to reflect this. See screenshot of a working blueprint bellow.
Also, there seems to be a timing bug in the MarkerTracker component where it won't work if it is initialized too early. We'll work with the team internally to get this resolved. For now, I was able to work around this by adding a short delay before creating the component. (see screenshot)
Finally, I noticed that the marker tracker component doesn't entirely clean up properly if you use the VR preview multiple times when running in the UE editor. This may be benign, but we'll likewise work with the team to resolve this. If you have trouble when running multiple times in the editor, we should be able to provide a workaround if necessary.
thanks for the great info and quick reply. I made all these changes above and Im still having issues detecting the aruco marker in VR preview. Just to double check what blue colored array input type do you have in your "MT RESULTS" custom event. I am using "Magic Leap MarkerTracking Marker". see pic
Also I simply went to this website printed off the first marker that appears with default settings. Online ArUco markers generator
Also does this code snippet belong in your pawn or a separate "tracker Blueprint" I have tried both and no luck.
I don't think it should really matter if it's on your pawn or another blueprint. I tested with it on another actor in the scene.
Also, I used that same website to generate a marker.
Btw- just as a quick test to check that it works at all, you could try opening up the examples project and run the marker tracking level.
I believe that's the type.
Btw, to be sure, and easy way to create the event is to drag a connection off of the Bind to result delegate node and click Add Custom Event... under Add event
I checked all those, still no luck. I tried the ML example tracker level also and that didnt work either. The status screen on sample level says "No Active Detectors" when I go to status
Any other ideas? Not sure if this helps but it reads the QR code on ML3 Hub to do remote connect.
Okey, well being able to scan the remote render pairing QR code would validate that the tracker at least works on your ML2.
I just tried running the v1.6 example project with the commercial release of UE 5.4.4 from the Epic Games Launcher and it does work for me.
Note the tracker doesn't start as soon as you load the example level. To start the tracker in the example level, you have to navigate to the "Controls" page in the dialog and press the Apply Configuration button. (even though the default config settings are probably what you would want to use)
So some good news, I got it to read the Aruco code now, but it doesnt do anything after that. I have a print string in UE for the and just returns the cords as 0,0,0 when I move the Aruco around different positions. Also in the Examples project file under the stats window of the aruco it also reads it but gives cords as 0,0,0 . Error 0.001.
Any ideas. I was hoping the debug cord system would snap to the paper code, and move along with the paper. But the debug cord it just sits at 0,0,0.
Hm, sorry, that is odd. Yes, the debug coordinate system should follow the marker around if things are working correctly.
The example level will render a coordinate system on top of the marker as well. I would guess that you see the coordinate system stuck at 0,0,0 in the example level as well right?
Would you have a way to share your UE project? I'd be happy to try running it on my machine and see if I get the same behavior.
Additionally, I think it might be worth trying a couple of other samples to rule out something with your device. It might be good to try a native openxr app that uses the marker tracker, as well as a windows openxr app over streamed using remote render that uses the tracker but not Unreal
I'll put a couple of samples together that you can try running in the mean time.
Actually, one thing just for sanity~ I'm sure you've tried this, but you could try rebooting your pc (to quit Unreal and ensure that the remote render service is restarted for good measure). Then, open the project, and the very first time that you try using the VR preview in Unreal after opening the editor, see what happens when you track a marker. I noticed that I think the marker tracker component may not clean up everything when you start/stop in the UE editor (it wouldn't be an issue when running the app outside the editor).
I sent the project over to my coworker, and he tried it with magic leap exact same results. Both in my proj and the example project, using commercial UE5.4.4. The file is really big 700 megs zipped, so im not sure how to send it to you. This is the error code I get in the example file...
I will try to package the windows exe... see what I get and reboot my machine
I just tried it again and it worked okey. I just sent you a copy of my test project. It should have kicked you an email, but I'll dm you a direct link since this thread is public. The windows folder has a the app packaged as an exe. It just renders a debug axis on top of Aruco markers with the default tracker settings. It should show an axis on top of the default marker you see on Online ArUco markers generator
Hi Adam, thanks for sharing your proj with us. I tried it both the packaged exe and the UE proj file both with same results. It will draw the debug cord only 0,0.0 when Aruco is recognized. It will not place the debug in space on top the aurco code. Its really weird this works for you guys, but not for us. Just to confirm, you are using the following:
UE Commercial build 5.4.4
Magic Leap Hub 3, with latest 1.6 Unreal plugins
Remote Rendering
Aruco print out with default setting from: Online ArUco markers generator
Hello, np. Yep, I was using all of the same versions. Latest remote render service & viewer (v1.15.131).
It's interesting that you and your coworker both had the same issue. If you reproduced the same behavior with the exe that I sent you, then that would eliminate any differences in your toolchain. It's likely something with your specific environment. I'd like to validate that the tracker works as expected on your device locally and that markers are still detected, but not tracked when using remote render in another app. I'll put a couple of apps together to validate this without using Unreal.
If remote render works at all and you get basic headpose and rendering, then that would largely eliminate network concerns, but just for context, what sort of network are you using for testing? Just a wifi router connected to a pc or something more complex? Also, just to confirm, when you first launched the remote render viewer app on your ML2, did you allow all permissions?
In the meantime, if you want to debug further, I'd be curious if there were any errors in various logs-
UE log
logcat (to see if anything was emitted by the remote viewer, you can monitor it live but running adb logcat if you're familiar with adb or grab whatever was recently logged by generating an adb bugreport
You can also try launching the remote render service manually from a command line to watch for any shell output while it's running (to do this, quit ML hub, kill windrunner-service.exe in task manager, open a powershell prompt, and run "C:\Program Files\Magic Leap Remote Rendering\bin\windrunner-service.exe"). Pairing information will have been cached on your ML2 and it should connect simply by opening the remote viewer app on your device, no need for ML hub.
The remote render page in ML hub also has an option in the ... menu to grab diagnostic logs and there's a script in the app install directory.
I sent you a native sample apk that you can try running on your devices to verify that the tracker is at least working correctly. Note that the sample is configured to detect 5x5 Aruco codes by default. The generator on chev.me is set for 4x4 by default.
It might take a bit more effort to get a sample together that you can try with remote render that doesn't depend on Unreal, although I suspect that we can probably work with what we have. If the native sample works for you, then we can move to trying to debug what's wrong with remote render.
I know it's not ideal, but we've been trying to keep support to the forums for now. First thing would be to check the UE log for any error output there. If an OpenXR function call failed, then you would likely see some error output in the Output Log in the UE editor.
Another thing that you could try would be using remote render over a usb connection, to eliminate the possibility of any sort of network concern. Low confidence that will get it working, but it's fairly quick to do and would eliminate some difference in your environment vs mine. We have some instructions on the dev portal-
So side note first: talking with my team lead (Patrick Oneill) about this issue. He had the same problem as both me and my teammate and the aruco code returning 0.0.0. He was working with Thoufeeq Ahamed on this same topic about 6 months ago. He got busy with other projects and never able to resolve it. He got as far as making it work in Android build but not windows package or UE editor. Not sure if that helps you at all, but just wanted you to know that’s 3 of us that cant get it to work on our end. Im thinking it has to be a setting somewhere that causing conflict… The apk works great when loaded on headset. So its not the headset itself. When your using UE commercial 5.4.4 do you have your plugins installed on the project level or the engine level? We put them in proj level. Maybe environment variable issue?
I have loaded your MLRRMarkerTEst proj in UE and this is the UE print log I get filled with errors while looking at the Aruco in editor runtime: Note I have a print string in the blueprint that calls the location of the break marker, if you see cords in the log below. Hope the log has some insight
