Getting ScreenSpace Coordinates from World Coordinates

Unity Development
,
21

Have you tested the script located on this page? Intrinsic/Extrinsic Parameters | MagicLeap Developer Documentation

22

Yes, the script on this page does 2D screen point to world position conversion. I am doing the opposite (world position to 2D screen point) and facing the issue described above.

23

Ahh I understand, unfortunately we do not have an example for this function but this calculation is not specific to Magic Leap as the RGB camera is a standard pinhole camera. In the meantime I have submitted your request for this type of sample.

Are you using the mixed reality or CV camera stream?

Here is a sample script, note I have not tested it thoroughly but can provide some guidance on the functions required to undistort the image :

using UnityEngine;
using UnityEngine.XR.MagicLeap;

public class CameraUtilities
{
    /// <summary>
    /// Converts a 3D world position into a 2D screen pixel coordinate.
    /// </summary>
    /// <param name="icp">Intrinsic Calibration parameters of the camera.</param>
    /// <param name="cameraTransformMatrix">Transform matrix of the camera.</param>
    /// <param name="worldPoint">3D world point to be converted.</param>
    /// <returns>The screen pixel coordinates corresponding to the given world space position.</returns>
    public static Vector2 ConvertWorldPointToScreen(ICamera.IntrinsicCalibrationParameters icp, Matrix4x4 cameraTransformMatrix, Vector3 worldPoint)
    {
        // Inverse the camera transformation to bring the world point into the camera's local space
        Vector3 localPoint = cameraTransformMatrix.inverse.MultiplyPoint3x4(worldPoint);

        // Project the local 3D point to the camera's 2D plane
        Vector2 cameraPlanePoint = ProjectPointToCameraPlane(icp, localPoint);

        // Convert camera plane coordinates to pixel coordinates
        Vector2 pixelCoordinates = ConvertCameraPlanePointToPixel(icp, cameraPlanePoint);
        return pixelCoordinates;
    }

    /// <summary>
    /// Projects a point from 3D space onto the camera's 2D plane using the camera's intrinsic parameters.
    /// </summary>
    /// <param name="icp">Intrinsic Calibration parameters of the camera.</param>
    /// <param name="point">The point in the camera's local space.</param>
    /// <returns>The point on the camera plane.</returns>
    private static Vector2 ProjectPointToCameraPlane(ICamera.IntrinsicCalibrationParameters icp, Vector3 point)
    {
        // Normalize the point by the depth to project it onto the camera plane
        Vector2 normalizedPoint = new Vector2(point.x / point.z, point.y / point.z);

        // Apply the camera's intrinsic parameters to map the normalized point to the camera plane
        Vector2 cameraPlanePoint = new Vector2(
            normalizedPoint.x * icp.FocalLength.x + icp.PrincipalPoint.x,
            normalizedPoint.y * icp.FocalLength.y + icp.PrincipalPoint.y
        );

        return cameraPlanePoint;
    }

    /// <summary>
    /// Converts a point from the camera plane to pixel coordinates.
    /// </summary>
    /// <param name="icp">Intrinsic Calibration parameters of the camera.</param>
    /// <param name="point">The point on the camera plane.</param>
    /// <returns>The corresponding pixel coordinates.</returns>
    private static Vector2 ConvertCameraPlanePointToPixel(ICamera.IntrinsicCalibrationParameters icp, Vector2 point)
    {
        // Convert the camera plane point to pixel coordinates by accounting for the image dimensions
        Vector2 pixelCoordinates = new Vector2(
            (point.x - icp.PrincipalPoint.x + icp.Width / 2),
            (icp.Height / 2 - (point.y - icp.PrincipalPoint.y))
        );

        return pixelCoordinates;
    }
}
1 Like