Extended reality covers virtual reality, which replaces the world with a rendered one, and augmented reality, which adds rendered things to the real one. Both must track where your head and hands are and redraw fast enough that your body believes it. Comfort is a hard technical limit, not a detail.
Key concepts
- Tracking: finding the pose of the head and controllers from inertial sensors, cameras and markers.
- Stereo rendering: one image per eye, with lens distortion corrected, at a high and steady frame rate.
- Latency: the delay between a movement and the image that answers it; too much causes discomfort.
- Spatial mapping: understanding surfaces and depth in the real room so virtual objects sit on them and hide behind them.
- Interaction: hands, gaze, controllers and voice, in three dimensions, with real scale.
Vocabulary
- 6DoF: six degrees of freedom, three of position and three of rotation.
- SLAM: building a map while tracking position inside it.
- Reprojection: warping the last frame to the newest head pose to hide small delays.
- OpenXR: the open standard API between XR applications and devices.
What to study, in order
- Rotations: Euler angles, rotation matrices and quaternions, and why gimbal lock happens.
- Render a scene in stereo on a screen, with a camera that follows the mouse.
- Build a small app through OpenXR or a game engine's XR support, and read the frame timing.
- Measure frame time and learn what dropped frames feel like; budget for it.
- Design an interaction in 3D and test it on people for comfort, not just for function.