HoloLens 2 Knee Replacement Surgery Guide
Mixed reality training app that walks medical students through total knee replacement in 3D, adapting to their decisions in real time. Domain: Medical training.
The challenge
Med students learning total knee replacement lack a spatial, interactive rehearsal environment that adapts to their decisions in real time.
The approach
HoloLens 2 / MRTK2 training app with an integrated Claude-powered cognitive agent over a custom surgical knowledge base, delivering step-by-step procedural guidance and live feedback on technique.
About this project
An immersive mixed reality training application for medical students learning total knee replacement. Built with Unity3D and MRTK2 for HoloLens 2, this interactive guide features AI-powered procedural assistance, offering real-time feedback on technique and decision-making. Provides step-by-step visualization of surgical procedures in 3D space, dynamically adapting to user performance and queries through an integrated cognitive AI.
Role
Lead developer. Built the HoloLens 2 application end to end - spatial interaction, procedural flow, and the Claude-backed guidance agent over the surgical knowledge base.
Outcome
10 min saved per training session
Technology and methods
- HoloLens 2
- Unity3D
- MRTK2
- Claude API
- RAG
Related work
- Mixed Reality Naval Maintenance Trainer - Gaze-driven HoloLens 2 trainer for F-18 maintenance procedures, with an agent pipeline that flags struggling trainees to supervisors.
- 3D Engine Bay Scanning & Tracking - Full pipeline for turning a physical engine bay into a tracked XR experience: scan, optimize, register, and overlay on the real hardware.
- Live IoT Data in Mixed Reality - HoloLens 2 app that pairs a virtual drone with live MQTT sensor streams, showing environmental telemetry beside the physical asset.