Mixed Reality Sand Table

MVRsimulation’s mixed-reality Sand Table provides a shared 3D visualization tool for realistic geospecific mission planning/rehearsal, classroom-deployed tactics training, real-world mission planning, and After Action Review (AAR). The mixed-reality Sand Table allows commanders, instructors, trainees, and students to interactively plan and review training missions in a collaborative, mixed-reality setting.

Traditional military sand tables consist of 2D maps, aerial photographs or sand molded into 3D representations of terrain, upon which tokens or cards are placed to represent friendly and enemy forces ahead of military actions. The MVRsimulation Sand Table replaces this physical system with a mixed-reality 3D ‘table’ upon which the virtual battlespace is rendered by Virtual Reality Scene Generator (VRSG) in its geospecific, round-earth 3D format. This allows users to understand how the terrain, elevation, and physical features of the landscape, such as mountains, valleys, trees and waterways will impact planned maneuvers; along with cultural and man-made features such as buildings, dense urban areas, roads, bridges, dams or other cultural infrastructure. Users can plan, enact, and review training scenarios while directly interacting with cultural objects in VRSG or by the use of any DIS-based Semi-Automated Forces (SAF) commercial or government off-the-shelf (GOTS) software.

Users are immersed in the virtual sand table via the use of Varjo mixed-reality headsets. The Varjo headsets provide the mixed-reality environment that allows the wearer to freely explore the 3D visualization of the scenario and geospecific 3D terrain. Mixed-reality provides real-world pass through for direct interaction with other observers in the real-world. Users can also operate physical emulated military equipment in the real world and simulated military training packages such as ATAK via the Special Warfare Assault Kit (SWAK) without the need to remove the headset. An instructor can interact directly with a student in the mixed-reality sand table while streaming the visuals of both users to an external monitor for classroom instruction. The instructor stays immersed in the sand table while also being able to see the other students in the classroom past the edge of the virtual sand table.

Each user has a Valve Index Controller to control their own unique view within the sand table. The controllers allow each user to independently traverse the terrain to evaluate specific views of the 3D virtual world. Users can zoom in and out with the controllers to give them vantage points from a close up evaluation of a single entity to a top-down view of the entire battlefield. The Valve Index Controllers also provide a virtual pointer used to point to specific elements in mission planning or AAR and to direct other participants’ attention to a particular location or entity.

During Sand Table use, the participant’s eye gaze is captured by the Varjo headset and depicted as a colored frustum allowing each user to see where all participants in a scenario are looking. The colored frustum is valuable for student training as it allows the instructor to verify in real-time if the student is looking at the correct location or entity. Other Varjo headsets connected to the DIS-based network can output their eye gaze tracking from VRSG. Eye-gaze tracking from role players in AAR is critical to remove ambiguity from the question of where a student’s attention was focused throughout the mission.

The network and LVC-ready Sand Table can be used as a stand-alone training and evaluation asset. It can also be integrated with any DIS-based training simulator such as MVRsimulation’s Deployable Joint Fires Trainer (DJFT) or Part Task Mission Trainer (PTMT). VRSG listens for DIS entities on the network and displays them as 3D models in the virtual sand table. DIS recording capabilities facilitate recording and playback of training sessions. During playback the instructor has control of the recording allowing them to pause the recording for all users to evaluate the full 3D virtual world including the geospecific terrain, 3D entities, line of sight, and student eye gaze.