Eye Tracking
An important aspect of a pilot’s simulated mission training is the ability to assess and discuss the air-to-air engagement in an after-action debrief so as to fully understand what lead to the outcome.
The real-time eye-tracking built into the Varjo XR-1 and XR-3 mixed-reality headsets adds a critical dimension to this normally subjective process. The wearing of this headset during a training mission helps provide quantitative data for the pilot’s after-action debrief by enabling reviewers to see where the pilot’s head and eye movement were during the simulated air engagement.
To take advantage of the eye-tracking technology in Varjo headsets, MVRsimulation has developed the means to visualize the gaze of the pilot. This feature is available in VRSG version 7. During a simulated flight, VRSG can track the pilot’s head position and orientation within the cockpit simulator, track the gaze vector using the Varjo device’s pupil tracking functionality, and then depict the gaze of each eye independently as a color-coded 3D cone. At the end of the training mission, VRSG can export this data via DIS as a PDU log.
During after-action playback, VRSG visualizes the pilot’s head position, orientation, and gaze vector over the events of the training session. From a tactical perspective, the eye-tracking playback can help identify missed moments of attention to instrumentation or the direction of an important activity. And could be useful for beyond visual range systems management review or for critiquing basic fighter maneuvers.
Fast-jet pilot training is just one example of eye-tracking with VRSG coupled with a Varjo XR headset. This setup can be used in any type of mixed-reality application.
MVRsimulation VRSG real-time scene visualizing the gaze of the wearer of a Varjo XR-3 mixed-reality headset through the collection of eye-tracking data. The scene features a JTAC entity in VRSG's First Person Simulator (FPS) mode on a rooftop located on Afghanistan 3D terrain.
Among the options for configuring VRSG with a Varjo XR system is foveated rendering. Foveated rendering increases the performance of rendering the entire scene in the headset. This option enables the GPU to render the scene at full fidelity in the area that the headset wearer is looking, while reducing fidelity in the peripheral areas of the scene.
In addition to recording the eye-tracking visualization for after-action playback, the VRSG session with the eye-tracking visualization can also be monitored by instructors in real time during the simulated engagement, to further enhance understanding of the nature of the training engagement.