# Implementing Yolov8 model in Unity Magic Leap 2 project

**URL:** <https://forum.magicleap.cloud/t/implementing-yolov8-model-in-unity-magic-leap-2-project/3707>\
**Category:** Unity Development\
**Tags:** development, unity, machine-learning\
**Created:** [June 27, 2024, 3:03pm UTC](https://forum.magicleap.cloud/t/implementing-yolov8-model-in-unity-magic-leap-2-project/3707 "2024-06-27T15:03:33Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![uros.mitrovic](https://avatars.discourse-cdn.com/v4/letter/u/b2d939/32.png) [@uros.mitrovic](https://forum.magicleap.cloud/u/uros.mitrovic)\
**Post date:** [June 27, 2024, 3:03pm UTC](https://forum.magicleap.cloud/t/implementing-yolov8-model-in-unity-magic-leap-2-project/3707/1 "2024-06-27T15:03:33Z")

</div>

I'm seeking to integrate object detection into an existing Unity application for Magic Leap 2. I'm currently using Barracuda as my machine learning library along with a custom-trained YOLOv8 ONNX model. While I've successfully conducted inference both in Unity and on the device, I'm facing challenges in interpreting the outputs.

Could anyone recommend a Unity-supported package for inferencing with YOLOv8? Alternatively, is there a more effective approach to implementing object detection using a trained YOLO model in ONNX format?

@sfernandez, I would greatly appreciate it if you could take a look at this topic. Thank you for your assistance and our past collaborations! 😊

---

<div class="post-metadata">

**Author:** ![localjoost](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/localjoost/32/1829_2.png) [@localjoost](https://forum.magicleap.cloud/u/localjoost)\
**Post date:** [June 28, 2024, 5:03am UTC](https://forum.magicleap.cloud/t/implementing-yolov8-model-in-unity-magic-leap-2-project/3707/2 "2024-06-28T05:03:53Z")

</div>

Does [this](https://localjoost.github.io/Running-a-YoloV8-model-on-a-Magic-Leap-2-to-recognize-objects-in-3D-space/) help?

Have fun with it, let me know what you think.

---

<div class="post-metadata">

**Author:** ![alain.birchmeier](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@alain.birchmeier](https://forum.magicleap.cloud/u/alain.birchmeier)\
**Post date:** [June 28, 2024, 5:50am UTC](https://forum.magicleap.cloud/t/implementing-yolov8-model-in-unity-magic-leap-2-project/3707/3 "2024-06-28T05:50:53Z")

</div>

I would recommand to check [Unity Sentis: Use AI models in Unity Runtime | Unity](https://unity.com/products/sentis) for ONNX format.

---

<div class="post-metadata">

**Author:** ![uros.mitrovic](https://avatars.discourse-cdn.com/v4/letter/u/b2d939/32.png) [@uros.mitrovic](https://forum.magicleap.cloud/u/uros.mitrovic)\
**Post date:** [July 1, 2024, 10:01am UTC](https://forum.magicleap.cloud/t/implementing-yolov8-model-in-unity-magic-leap-2-project/3707/4 "2024-07-01T10:01:42Z")

</div>

I tried using that solution, but I received a strange output. The values I'm extracting from the model are center, top-left, bottom-right, and confidence. All these values seem small, so I'm unsure how to convert them into bounding boxes. Additionally, the confidence values should be within the range of 0 to 1, but I'm getting values above 1.

![image](https://us1.discourse-cdn.com/flex019/uploads/magicleap1/original/2X/7/7b4ca6ed135fbdf429f5c2eef359d8fa484ba761.png)

---

<div class="post-metadata">

**Author:** ![localjoost](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/localjoost/32/1829_2.png) [@localjoost](https://forum.magicleap.cloud/u/localjoost)\
**Post date:** [July 1, 2024, 1:25pm UTC](https://forum.magicleap.cloud/t/implementing-yolov8-model-in-unity-magic-leap-2-project/3707/5 "2024-07-01T13:25:18Z")

</div>

Yeah, I am not quite the Yolo expert. It feels like you putting the wrong values in or reading the wrong values out of a tensor. But that is more of a hunch. What does Netron display about inputs and outputs?

---

<div class="post-metadata">

**Author:** ![uros.mitrovic](https://avatars.discourse-cdn.com/v4/letter/u/b2d939/32.png) [@uros.mitrovic](https://forum.magicleap.cloud/u/uros.mitrovic)\
**Post date:** [July 1, 2024, 1:59pm UTC](https://forum.magicleap.cloud/t/implementing-yolov8-model-in-unity-magic-leap-2-project/3707/6 "2024-07-01T13:59:33Z")

</div>

This is the Netron results:  
 ![image](https://us1.discourse-cdn.com/flex019/uploads/magicleap1/original/2X/1/1ccea22314bf86a5a08acf30fdfe173dde77ce58.png)

Data are read as Barracuda.Tensor. This is the exact code for reading data:

```auto
Center = new Vector2(tensorData[0, 0, 0, 0], tensorData[0, 0, 0, 1]);
Size = new Vector2(tensorData[0, 0, 0, 2], tensorData[0, 0, 0, 3]);
TopLeft = Center - Size / 2;
BottomRight = Center + Size / 2;
Confidence = tensorData[0, 0, 0, 4];

```

As inputs I pass the texture of image in resolution 640x640. Basically in the same way preprocessing is done in the example.

---

<div class="post-metadata">

**Author:** ![localjoost](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.magicleap.cloud/localjoost/32/1829_2.png) [@localjoost](https://forum.magicleap.cloud/u/localjoost)\
**Post date:** [July 1, 2024, 6:13pm UTC](https://forum.magicleap.cloud/t/implementing-yolov8-model-in-unity-magic-leap-2-project/3707/7 "2024-07-01T18:13:07Z")

</div>

Like I said, I am not a Yolo expert and I have no idea if you are actually using more of my code or not. Have you made sure the model is exported with opset 9?

---

<div class="post-metadata">

**Author:** ![tchiang](https://avatars.discourse-cdn.com/v4/letter/t/0ea827/32.png) [@tchiang](https://forum.magicleap.cloud/u/tchiang)\
**Post date:** [July 3, 2024, 5:26pm UTC](https://forum.magicleap.cloud/t/implementing-yolov8-model-in-unity-magic-leap-2-project/3707/8 "2024-07-03T17:26:24Z")

</div>

In YOLO v8, the center and size are normalized by resolution. So when convert to topleft ro bottomright, you have to multiply center and size by resolution.  
Here is a Matlab script I wrote to convert topleft/bottomright to center/size, you can reverse it to get what you want which are x1, y1, x2, y2:

```auto
%reso is the resolution, e.g. [640 480]
%box is the bounding box [topleftx, toplefty, width, height]
% x, y are the normalized center, w, h are normalized width and height
% x1, y1, x2, y2 are coords of topleft/bottomright (non normalized)
function [x,y,w,h] = convert2yolo(reso, box)
    dw = 1./reso(1);
    dh = 1./reso(2);
    w = box(3);
    h = box(4);
    x1 = box(1);
    y1 = box(2);
    x2 = x1 + w;
    y2 = y1 + h;
    x = (x1+x2)/2;
    y = (y1+y2)/2;
    x = x*dw;
    w = w*dw;
    y = y*dh;
    h = h*dh;
end

```

---

<div class="post-metadata">

**Author:** ![uros.mitrovic](https://avatars.discourse-cdn.com/v4/letter/u/b2d939/32.png) [@uros.mitrovic](https://forum.magicleap.cloud/u/uros.mitrovic)\
**Post date:** [July 4, 2024, 1:31pm UTC](https://forum.magicleap.cloud/t/implementing-yolov8-model-in-unity-magic-leap-2-project/3707/9 "2024-07-04T13:31:39Z")

</div>

I managed to solve this issue by flipping the image pixels

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/magicleap1/original/1X/bd9b737a4d45ea243aa7eb177d60a773b806899a.svg) [@system](https://forum.magicleap.cloud/u/system)\
**Post date:** [July 19, 2024, 1:31pm UTC](https://forum.magicleap.cloud/t/implementing-yolov8-model-in-unity-magic-leap-2-project/3707/10 "2024-07-19T13:31:54Z")

</div>

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.