Augmented reality (AR) is allowing radiologists and surgeons to plan procedures using image-derived holograms. Widespread intraoperative use may not be far behind.
AR enhances the physical environment by overlaying objects in the user’s field of vision with computer-generated images. Unlike virtual reality, AR does not replace the real-world environment with an artificial one — rather, it supplements the user’s perception of the surrounding world with additional information. That makes AR ideal for use in medicine.
Utah-based Novarad was a pioneer in medical AR, and its OpenSight software was the first AR solution to receive FDA 510(k) clearance for presurgical planning. OpenSight pairs with Microsoft’s HoloLens headset to create 3D holograms from a patient’s CT or MRI studies. This technology superimposes the holograms on the patient, allowing radiologists and surgeons to see inside the body without looking at a screen. QR-coded stickers placed on the patient’s body sync with imaging studies and allow OpenSight to precisely register the patient in space. That ensures the holograms, which display internal structures in true-to-life size, align precisely with the patient’s anatomy.
“OpenSight takes images and turns them into holograms that project onto patients, allowing radiologists and surgeons to see patients and see into them at the same time,” says Wendell Gibby, MD, a diagnostic neuroradiologist and Founder and CEO of Novarad. “With this technology, clinicians can identify anatomy, localize tumors and plot how to avoid healthy structures by planning surgeries and performing them virtually instead of making decisions based on 2D images on a screen or 3D printed models.”
OpenSight has a variety of applications for surgical planning and performance, with the latter currently taking place under IRB protocols while Novarad pursues FDA clearance for intraoperative use, according to Dr. Gibby. At The George Washington University Hospital, clinicians have used OpenSight to assist with inserting metal hardware to stabilize fractured ribs. A key to the surgery, known as rib plating, is making incisions directly above the fractured ribs. The current practice of using a CT scan to guide where to cut can lead to imprecise incisions. A clinical trial at the hospital found that when surgeons used OpenSight to turn patients’ CT images into holograms and then used those to guide where to open the patients, their incisions were far more accurate than when they used CT alone. In rib plating procedures, Dr. Gibby says precise incision placement can lead to smaller incisions and faster surgery and recovery.
Other hospitals and universities in the U.S. and Canada, including Memorial Sloan Kettering Cancer Center, Nicklaus Children’s Hospital, Miami University and McGill University, have used OpenSight for orthopedic, neurosurgical and cardiovascular surgery applications. Dr. Gibby has used OpenSight to plan and perform interventional radiology procedures, including percutaneous discectomy and cryosurgery.
“OpenSight presents radiologists with a tremendous opportunity to be part of a team that can help plan and execute more precise surgeries,” Dr. Gibby says. “We’ve gone through a series of revolutions in imaging from X-rays to digital radiology to 3D imaging. Now, I think the ability to migrate images from a screen to the patient’s body will be transformative for both radiology and surgery.”
Innovating the Future
Novarad isn’t the only player in medical AR. Philips Healthcare worked with Microsoft to pair the former’s Azurion software with HoloLens to create an AR solution for minimally invasive procedures. Interventional radiologists at Cincinnati Children’s Hospital Medical Center have used Philips Healthcare’s AR technology to practice advancing needles through simulated patients and rehearse spine and thoracic procedures with surgeons.
Individual innovators are also helping to advance AR in medicine. At UC San Francisco, Jesse Courtier, MD, Associate Clinical Professor of Pediatric Radiology and Co-Director of the UCSF Center for Advanced 3D Imaging, started exploring AR’s potential because he wanted to create 3D models that would allow his surgical colleagues to exploit imaging studies to their fullest. He developed a software application called RadHA (Radiology with Holographic Augmentation) that creates 3D holographic models from CT and MRI studies.
“I’m focused on helping surgeons with presurgical planning and determining whether an operation is feasible,” Dr. Courtier says. “Sometimes, a surgeon will start a procedure and realize it’s much more complex than previously thought, so we’ve been having surgical planning conferences where we use these holographic models to help surgeons decide whether a procedure can be done and, if so, what approach, incisions and hardware to use.”
Currently, through his startup, Sira Medical, Dr. Courtier uses RadHA to create image-derived holographic models that surgeons can view using a smartphone and a handheld device called a Merge Cube. When the Merge Cube syncs with the app, the user can “hold” the holographic model on the cube and examine it in 360 degrees using a smartphone as a viewfinder. Recently, Dr. Courtier modeled a complex ankle fracture for a pediatric orthopedic surgeon, and the two examined it in conference.
“Rather than scanning through CT studies, surgeons can use a holographic model to peel back layers of a structure and figure out how to approach a procedure,” Dr. Courtier says. “That’s helpful for them, and using a phone-based app is efficient and intuitive.”
Dr. Courtier’s holographic models may also save time. He conducted a preliminary, retrospective study that found using holographic models reduced elbow fracture surgery times by an average of 20% compared with standard practice. That may lead to less time under anesthesia for patients.