Mixed Reality Naval Maintenance Trainer
Gaze-driven HoloLens 2 trainer for F-18 maintenance procedures, with an agent pipeline that flags struggling trainees to supervisors. Domain: Aerospace maintenance training.
The challenge
F-18 maintenance trainees struggle with corrosion-detection procedures, and supervisors lack early signals when a trainee is falling behind.
The approach
HoloLens 2 training app with gaze-based interaction paired with an N8N + LLM agent pipeline that ingests xAPI data from a Learning Record Store, analyzes error patterns, and alerts supervisors for targeted intervention.
About this project
Advanced mixed reality training application for US NAVY maintenance procedures using HoloLens 2. Utilizes gaze detection for intuitive interaction. AI algorithms process xAPI performance data to generate predictive analytics on trainee proficiency, optimize training workflows, and recommend targeted interventions, leading to more efficient skill acquisition.
Role
Lead developer. Designed the gaze-based interaction model and built both the HoloLens 2 trainer and the xAPI analytics pipeline that drives supervisor alerts.
Outcome
+15% task accuracy / -20% training time
Technology and methods
- HoloLens 2
- Unity3D
- N8N
- xAPI / LRS
- LLM Agents
Related work
- HoloLens 2 Knee Replacement Surgery Guide - Mixed reality training app that walks medical students through total knee replacement in 3D, adapting to their decisions in real time.
- 3D Engine Bay Scanning & Tracking - Full pipeline for turning a physical engine bay into a tracked XR experience: scan, optimize, register, and overlay on the real hardware.
- Live IoT Data in Mixed Reality - HoloLens 2 app that pairs a virtual drone with live MQTT sensor streams, showing environmental telemetry beside the physical asset.