Hi @kulwik,
I went ahead and tried to reproduce this issue. It seems that the simulator creates illusions due to transforming a 3D projection onto a 2D screen and the ray's draw order. So when the ray is not hitting a UI element that can be interacted with, it turns red. It passes through the UI which makes it seem as though it's pointing at the UI element. However, in this example scene if the ray is red, it is not hitting any UI elements. I have changed the rendering mode to two-eyed in the app simulator to help demonstrate this.
In the screenshot above, the unity scene shows that the ray is passing through the non-selectable UI elements, but in the right eye view it seems like the ray is pointed at the 'X' button but it is actually behind it. If you move the controller around in the scene view of the app simulator by selecting it and changing the manipulation mode to "rotate."
After rotating the controller, I was able to get the ray to turn white and I was able to select the UI elements as shown below.
Let me know if this helps.
Best,
El