XR Training for Crisis Operations
Source: idw-online.de
Immersive Training for Emergency Responders
Emergency responders can now train for critical situations in immersive XR environments safely using Immergensim. It was developed through the EU-funded CORTEX² project.
The AI mentor, Maud, supports the team. The responders are new, but prepared through a virtual trial run that helps them make decisions under pressure and coordinate their responses. Inside the simulation, Maud watches over the team, supports their choices, provides resources, answers questions, and helps build the instincts they’ll need.
CORTEX² Platform
Researchers at the German Research Center for Artificial Intelligence (DFKI) created a flexible infrastructure under the CORTEX² project that enables humanitarian responders to conduct virtual crisis training. The immersive training system Immergensim was developed within the XRisis sub-project by XR Ireland with Action Contre la Faim (ACF), building on CORTEX²’s technological foundation.
CORTEX² was created because there was a lack of digital tools for tasks requiring physical presence, visual collaboration, or physical-gestural interaction. Traditional video conferencing had limitations, lacking solutions that transmitted presence, interaction, and real-time coordination in three-dimensional space. DFKI collaborated with European partners to develop CORTEX², a modular platform for cooperative extended reality that combines real-time communication, AI integration, and immersive learning spaces.
Key Features
CORTEX² provides an architecture designed for interoperability and customizability, offering a platform-capable infrastructure open for adaptation, expansion, and cross-sector use. Features include a real-time communication system, an avatarisation system, AI virtual conversation partners, an automated evaluation tool, and real-time modeling of real objects. An IoT server enables physical devices to connect to virtual scenarios.
Researchers used the Rainbow product from Alcatel Lucent Enterprise, a tool for video conferencing similar to Teams and Zoom. Functions of the Rainbow core were transferred into a functional framework for XR applications, providing a range of AI applications from a modular system of functions and tools.
Services provided by CORTEX² include:
- Rainbow CPaaS (Alcatel-Lucent): secure voice and video connections in XR rooms
- VCAA (DFKI): Video Compression Alternative Appearance - an avatarisation system
- CoVA (CEA): Conversational agents - AI-based virtual dialogue partners
- Summarisation Agent (Linagora): Automatic call summaries
- CORTEX IoT Server: Management and integration of IoT devices
- 3D reconstruction: Real-time 3D modeling of real objects
All services consider data protection, scalability, and technical resilience.
Immergensim in Action
The CORTEX² consortium created XRisis to address gaps in training humanitarian aid workers. Traditional approaches face limitations like high costs, accessibility, and realism, especially for local actors. XR Ireland and Action Contre la Faim developed Immergensim, an immersive simulation platform featuring low-threshold, interactive training formats on desktop computers or VR headsets, recreating real-life emergencies.
Immergensim’s training concept has three modules: an introduction guided by the AI mentor avatar “Maud,” a team decision-making simulation in a virtual crisis coordination center, and the implementation of emergency measures in simulated field conditions with AI support. The modular training units adapt to diverse humanitarian missions. Interactive elements and dynamic decision paths make the experience practical and measurable.
The system integrates the Rainbow communication service into the Unity environment, enabling voice calls and collaboration. The hardware-intensive VCAA component was replaced with a web-based avatar emulator using MediaPipe, WebRTC, and React. The system integrates CoVA and the Summarisation Agent to structure feedback. This makes Immergensim a simulation-based training solution, operable even in resource-constrained settings such as NGOs and field offices.
Field feedback confirms Immergensim’s potential. Participants valued the soft skills training and collaboration in the virtual crisis coordination environment. The experience gained is being used to refine the platform. Immergensim was awarded the Unity for Humanity Grant in 2025. The team is developing a low-code SimEx Builder to enable NGOs to design their own immersive training simulations. The system’s potential can be adapted for other sectors like safety training, healthcare, and education.