Back in the mid-1960s, rotary-dial telephones were the norm, music cassette tapes were brand new, and microwave ovens hadn’t made it into houses yet. That’s also when newly minted electrical engineer Thomas Furness joined a U.S. Air Force Lab and began developing what would become the first virtual reality (VR) systems, including the fighter-jet Super Cockpit that gives pilots 3D representations of avionics data, along with the ability to use voice commands, hand gestures, and even eye movements to control the aircraft.
Now 50-plus years later, Furness is known as “the grandfather of VR,” and he continues to push the limits of VR at the University of Washington’s Human Interface Technology Lab (HIT Lab) and sister HIT Labs in New Zealand and Australia. Here, Furness (pictured above) describes his role in the early rise of VR, how he transitioned his work from the military to the public sector—especially for medical applications—and what he thinks must happen before VR can realize its vast potential in health care.
The Rise of VR
In 1966, Furness had just finished his bachelor of science degree in electrical engineering and completed Reserve Officer Training Corps (ROTC) training at Duke University when he received a commission as an officer in the Air Force with an assignment to the Armstrong Laboratory, Wright Patterson-Air Force Base, OH, USA. “The Armstrong Lab was working on the human–computer interface, and my main job was to design cockpits for military aircraft, especially fighter airplanes,” he said. “In a typical airplane, we were seeing 75 displays and 300 switches, including 11 switches on the control stick and nine switches on the throttle—to be used by pilots who were flying at twice the speed of sound and pulling Gs at the boundaries of consciousness, while being fired upon by enemy aircraft.”
This exceedingly complex task of piloting had to be made more intuitive, and Furness saw a way to do it. “We had already started using very limited virtual images in head-up displays in cockpits, but I thought, why can’t we expand that picture and make it move as you move your head around? So I started to work on the technologies that would do this, such as a way to track helmet position and to project images onto the visor of the helmet so you could basically take all the instruments that you had in the cockpit and project them out into space,” he said. That also meant developing an array of sensors that could be aimed wherever the pilot was looking. “For flying at night, for instance, you would have a forward-looking infrared sensor you could aim in real time, so it was just like you were looking through the aircraft and into the night.”
Over the ensuing years of work, this culminated in the Super Cockpit—a cockpit you wear—according to Furness. “It projects a 3-D wide-field-of-view scene to your eyes; uses things like eye, head, and hand tracking; adds three-dimensional sound; and uses expert systems and artificial intelligence (to create) a deeply-coupled system between the human’s sensory end organs—visual, auditory, and tactile—and the machinery.” Put another way, the Super Cockpit provided natural, intuitive interfaces that allowed a pilot to control the fighter jet almost as if it were an extension of his body.
This was the foundational technology behind VR.
Turning Point
The Super Cockpit went from military secret to media sweetheart in 1986, when the Pentagon decided it was time to publicize the work of Furness’ group. After a big splash on the CBS Evening News, media outlets from around the world came calling, and the subsequent coverage soon spawned a flurry of phone calls from people who wanted to know if the technology could be adapted for other purposes ranging from allowing surgeons to see inside patients during operations, to helping firefighters navigate through and communicate in smoke-filled buildings, Furness recalled. “That’s when I realized we were onto something really big, something that was transformative in terms of how we can couple humans to these computational engines in an immersive way.”
Furness ultimately decided that the best avenue for advancing VR applications was outside the military, and in 1989, he left the Air Force to establish the HIT Lab at the University of Washington, where he could develop transitional technology, generate patents, and spin off companies to get VR into the mainstream. The lab was highly interdisciplinary, drawing students from engineering to medicine, and psychology to the fine arts. “I would put these students in clusters of four and I’d seed them with a problem to solve. It was unbelievable. We had four patents a week from these kids because they had different perspectives,” he said.
One of the HIT Lab’s earliest products was developed in collaboration with a company called Virtual Vision, and went on the market in 1993–1994 as the very first commercial virtual display. “It looked like ski glasses or almost like a HoloLens does now, and you’d see a 1-meter virtual image at about 3 meters away. It was like your own personal headworn theater,” Furness described. Although not the commercial success he had hoped, the headsets were a surprising hit among one health care sector: dentists. “They were putting them on their patients, who would sit in the chair, select a movie, and while they were watching this movie in virtual space, the dentists could go about inflicting pain on them, but the patients were no longer complaining about it,” he said. “Dentists also reported that children were connecting Nintendo games to the VR headset, and becoming so engrossed in playing the game that they didn’t notice any pain at all,” Furness said.
Pain Control and More
The idea that VR could distract a person from feeling pain was a watershed moment. Furness had just hired psychologist Hunter Hoffman to work in the HIT Lab, and they started investigating the implications. Hoffman began collaborating with clinicians at the Children’s Hospital in Seattle, Washington, who were caring for youngsters with leukemia. These children had to undergo extremely painful bone-marrow extractions as part of their treatment, Furness explained. “Normally the kids would scream when the needle went in, but when we’d put the headset on them and let them play Nintendo, they’d just say ‘ugh’ and keep playing the game. The doctors and nurses had never seen anything like this.”
That success led to additional studies of VR for pain therapy, potentially as an alternative to costly and often addictive opioids, as well as spinoff companies developing pain-control systems for clinical use (see also, “Virtual Reality Is Taking the Hurt Out of Pain”). “It has really taken off,” Furness said.
Beyond pain management, the family of HIT Labs is working on several other areas related to health care. A major emphasis is on VR training simulations. Examples include teaching medical residents how to perform various surgeries, including how to handle unusual problems that may crop up during an operation; helping nurses to visualize the transferal of germs from one surface to another, and to maintain a sterile surgical field; and to train staff in neonatal intensive care units so they provide the best of care for their tiny patients.
The HIT Lab is also now studying VR as an option for evaluating and treating individuals with dementia. Currently, doctors evaluate patients with subjective measures, Furness said, “But when we put a patient into a virtual world, we can measure everything: the head acceleration, velocity and position, eye movement, hand movement and tremor. VR can become an enormous diagnostic tool.” The researchers are also seeing improvements in memory and engagement as patients use VR, he added. “We can adapt what they’re experiencing as they get better, so we can actually take them back up the curve.” While it’s not a cure for dementia, he acknowledged, this can help to “keep the lights on as long as the body is working.”
Getting VR into the Clinic
With researchers and companies finding new uses for VR in the medical arena, and insurance companies now becoming interested in adding VR to the health care arsenal, why haven’t these systems become prevalent in hospitals and clinics in the United States? “It’s called Medicare and Medicaid. That’s really what’s holding us back,” Furness asserted. “I look at the work we’re doing using VR as intervention for people with dementia, and it works. But the problem is that only the rich people can afford it. We need physicians and clinicians to be able to prescribe VR as if it’s a medication, with a code for it.” At present, Medicare does not include VR within its Healthcare Common Procedure Coding System, nor does it reimburse for it, and Medicaid also does not recognize VR as an approved treatment.
Fortunately, Furness said, U.S. health-insurance companies are now beginning to see the economic advantages of VR in speeding recovery times, replacing at least some pain medications, and cutting medical costs. An example is the health-insurance giant United Health Group, which is now funding VR research at a HIT Lab-inspired company Virtual Therapeutics of Redmond, Washington. Furness believes this is exactly the type of collaboration that will help prove the great value of VR in health care, and hopefully get Medicare and Medicaid on board. “The technology is running fast with new displays, new tracking systems, new software, and the tools to develop content, so we’ve built this 500-horsepower engine, but the fuel for this engine ultimately has to come from Medicare and Medicaid,” he remarked.
Furness concluded, “It’s not about the technology anymore. It’s about our healthcare system and how we should really exploit what can be provided to us with the new tool we have in VR.”