Beads of sweat trickle down your forehead. As your heart races, the screen becomes more static. Though it’s just a video game, you feel imprisoned in a nightmare.
What makes it so real is how the video game responds to every physical change. On screen, in your virtual room, liquid pours in. If you panic, you will drown. Your only hope is to take deep breaths to calm yourself to clear the room and advance to the next level. The game, called Nevermind, is one of the latest attempts to respond to a user’s physical state and provide an interactive experience that, aside from being just plain fun, might also have beneficial uses (Figures 1 and 2). In this case, the game could help users become aware of and, ideally, master their emotions.
A host of unexpected consumer technologies are forming around the field of physiological and biometric tracking—which is the measurement of bodily functions like changes in heart rate, skin conductance, the pattern of eye movements or voice inflection, gait, and even brain waves. Step counters and smartphone fingerprint scanners are some of the more common devices, but now, a dramatic drop in cost and rise in accuracy of physiological trackers—especially heart-rate monitors—is resulting in interactive applications beyond the plethora of fitness apps. These new technologies aim to help users learn selfcontrol, monitor deadly disease, and keep bank accounts safe, to name a few.
Measuring heart rate is one of the simplest ways to detect a physiologically aroused state, which generally indicates stress, fear, or excitement. Phones, smart watches, and fitness bands typically pulse a light-emitting diode onto the wrist or fingertip and use photodiodes to record the light that bounces back (which will differ slightly as blood pumps by) to determine heart rate. (See the article “The Telltale Heartbeat”) Other technologies rely on precise movement measurements, skin sweatiness, or even cameras to detect states of physiological arousal. Erin Reynolds, the creator of Nevermind, (pictured right) says the speed at which heart-rate sensors and other body-tracking technologies have advanced is offering exciting possibilities for physiologically interactive games. “As tech like this becomes part of everyday life—and becomes even cheaper and more accessible—this barrier of entry will be drastically minimized in the coming months and years,” says Reynolds, who is the creative director of Flying Mollusk, based in Glendale, California.
Nevermind, for example, can be synced with everything from Bluetooth-enabled optical monitoring wristbands to Garmin chest straps that measure the heart’s electrical pulses. The Apple Watch, Reynolds hopes, is next. She has also integrated the game with Intel’s RealSense three-dimensional camera, which determines a pulse by detecting tiny fluctuations in skin tone (when your heart beats, a rush of blood moves beneath your skin, causing your face to subtly flush).
Regardless of the method of delivery, the player’s heart rate is sent in real time to the horror mystery game, as users make their way through surreal levels of a patient’s subconscious mind and find clues to uncover a trauma (Figure 3). At each level, the player experiences new barriers if his or her heart rate begins to climb. “People come away saying they feel more appreciative of stress management from the real-time feedback: if they take deep breaths, for example, they immediately see how it can calm them,” says Reynolds, who had more than 1,500 backers on Kickstarter for the game. “That mindfulness is a big win for us.”
Biofeedback training—becoming aware of physical responses to better control them— has a long history in therapeutic psychological and meditative practices. However, Reynolds says tools like wearable heart-rate monitors place biofeedback systems within game developers’ reach, with the potential to spread the technology to a mainstream audience.
“At the end of the day, we’re all trying to get this niche tech out there and find how can we best present it so people can engage and benefit,” says Reynolds, who is also exploring additional body tracking that could chart emotions more accurately. Despite the potential for more immersive and reactive games, there are some challenges to incorporating physiological tracking: aside from the additional cost the user must incur to get the heart-rate sensor, game developers need the software to support biofeedback data. These development tools have only recently started to emerge.
Amsterdam-based health care software company Happitech, for example, built a health technology platform to allow developers to use heart rate as a control for a phone app, whether for dating, chat, or games (the iPhone’s camera and flash optically record heart rate). While many smartphone apps can track heartbeat for fitness or meditation exercises, the company’s Skip a Beat app, released last February, claims to be the first to incorporate biofeedback aimed at teaching relaxation into an iPhone game. Users tap the screen to control a bird but get a better score if their heart rates stay between a specific range, which is most commonly achieved through deep breathing, peaceful visualizations, and other techniques. Games like these, proponents suggest, can be more engaging than traditional biofeedback exercises—and there may be clinical evidence to support the idea.
More Than Biofeedback
In some cases, researchers have already demonstrated the potential benefits of emerging body-tracking technology in a clinical setting: people with anger or anxiety issues, as well as those with epilepsy, could benefit from technologies that respond to their physiological states.
On a trip to Greece, Joe Gonzalez-Heydrich, a psychiatrist at Boston Children’s Hospital and associate professor at Harvard Medical School, was struck by a statue of Hercules battling enemies with a calm expression. Gonzalez-Heydrich, who treats children with severe anger, thought that it epitomized what he wanted his patients to achieve. “Learning to stay calm in the midst of a challenge like that would be truly beneficial,” he says, adding, “Better emotional control in preschool and kindergarten predicts a child’s success all the way through high school.”
When he looked into traditional biofeedback applications, he found they tended to teach kids to withdraw and think about something calming. However, detaching from the moment isn’t always practical. “I thought, if we had a game that could call on players to regulate emotions while rapidly reacting and thinking on their feet, that would be more helpful for learning, for example, how to stay calm in an argument,” says Gonzalez-Heydrich.
He partnered with Boston Children’s Hospital’s Jason Kahn, Ph.D., who creates new technology for mental health care and works with psychotherapist Peter Ducharme. The team developed an exploratory game called RAGE Control, in which players must shoot asteroids threatening Earth and avoid harming allies, but with a biometric twist: they lose the ability to fire if their heart rate goes up (Figure 4).
In a 2012 pilot trial of five sessions, the team found that 18 patients who played the game in conjunction with receiving standard care (like cognitive behavioral therapy) had lower anger scores than a control group of 19 children who received standard care. In an interesting adaptation, the team also developed a cooperative, two-player version to include parents.
“We observe a lot of parents and kids who are constantly escalating each other and have no tools to regulate it,” says Kahn. In this version, the child and parent play together as a team; if either player is upset or stressed, both players are penalized for the rising heart rate. “The game encourages the child to apply calming techniques to a frustrating interpersonal situation and solidify those techniques by explaining to the parent how to stay calm so the team can score points,” he says.
The research team saw powerful outcomes in a second trial comparing 20 children randomized to the game therapy and 20 whose therapy was assisted by a “sham” computer game. After ten sessions of playing a game together following psychotherapy, the group playing the therapeutic game saw a greater decrease in reports of aggression and disruptive behavior at home and in parental stress.
“This is just not biofeedback,” says Gonzalez-Heydrich. “Those technologies have been around for a long time. This is engagement: it requires users to learn to rapidly react and manage heart rate during the social challenges of working as a team.” Gonzalez-Heydrich, Kahn, and Ducharme recently founded Boston-based Neuro’motion to launch a consumer version for iPhone and iPad by the beginning of 2016. They are also looking into additional body tracking parameters and planning a larger clinical trial to see how long into the future the benefits of the game last.
Other biometric games based on clinical research also aim to tackle mental health. The PlayNice Institute and GainPlay Studio, based in The Netherlands, designed MindLight, a child’s game that uses the NeuroSky MindWave headset to provide electroencephalogram, or brain wave, biofeedback and is currently undergoing clinical trials to see whether it can help children manage and overcome anxiety symptoms.
Detecting Hard-to-Sense Threats
Aside from evidence-based, clinical therapeutic games that aim to confer healthy habits, new applications that track heart rate as well as other physiological readings in a wearable form could be used to save lives more immediately. A number of heart-rate detectors aim to sense issues related to the heart, but one wearable tracker may be able to detect an unexpected health threat: seizures.
Rosalind Picard, a Massachusetts Institute of Technology (MIT) professor who founded the field of affective computing (creating machines that recognize and respond to human emotional states), was developing wearable physiological sensors long before Fitbit or Apple Watch came onto the scene. (See the interview “The End of Seizures and Depression?”) Aside from heart rate, Picard captures and interprets patterns of change in skin conductance: when the body is physiologically stressed, the skin becomes more electrically conductive (akin to clammy or slightly sweaty skin). In her devices, small electrodes pass an imperceptible current through the skin to detect this conductance.
While testing her latest biometric sensor, Picard noticed something interesting: one signal from a user skyrocketed, going far higher than signals from people taking an MIT qualifying exam or trying to catch a flight. “I thought the sensor was broken,” says Picard. “But I learned the boy who had produced the signal was having a seizure at the time.”
On further examination, Picard found that her sensors were able to detect seizures deep in the brain. She commercialized the technology as a data-collecting wristband geared for research (the E4) and as a consumer product (Embrace) through the company Empatica (Figure 5).
“The goal is to provide a tool to prevent deaths from seizures,” says Picard. About 20–40,000 people in the United States die each year from sudden seizures, according to the Epilepsy Foundation. The U.S. Centers for Disease Control state that the exact cause of death is unknown in these cases but could result from an interruption in breathing.
“Most deaths don’t happen during the seizure but after, when motor movement can be suspended and someone pauses in his or her breathing,” says Picard. “By giving people a better alert device, you can help them have a little more freedom and peace of mind than they have today.”
“Other items on the market, such as optically based heart-rate monitors on smart watches, aren’t ideal for constant monitoring,” says Picard, as they take up a lot of battery life and are less accurate in predicting changes than skin conductance coupled with motion-based heart-rate detectors, which use a combination of accelerometers and gyroscopes.
Embrace, which was funded through IndieGoGo and launched at the end of 2015, aims to detect and alert a user or family to a seizure and provide whatever the doctor recommends for that patient (which could be instructions to prompt the patient to talk, administer medicine, or call 911). The company is in the midst of pursuing U.S. Food and Drug Administration approval for the devices.
“There’s no question biometric wearables are going to revolutionize health care,” says Picard, whose team is also testing whether the sensors can predict seizures or be useful for patients with other disorders, such as post-traumatic stress disorder and migraines. “Wearables are going to enable us to do continuous monitoring to detect lots of interesting situations and intervene to prevent problems.”
Unexpected Uses
While physiological tracking in games as well as mental health and disease applications will likely grow, the technology is also blooming in other realms. For example, retina scans and fingerprints have long been a staple of identity authentication, but Goode Intelligence, a British consulting agency for the biometric and banking industry, released a report in June 2015 forecasting that, by 2017, more than 1 billion users will access banking services through biometric authentication systems. Banks are exploring new forms of technology, including heart sensors like the Nymi wristband, that can identify a person’s unique echocardiogram to prevent fraud.
“There’s a need to raise the bar from an identity and authentication purpose but also to have easy-to-use solutions that work across a wide range of banking channels and on a wide variety of end points, including mobile and wearable,” says Alan Goode, founder of Goode Intelligence. “In terms of newer biometric technologies, we are at the beginning of the adoption curve. We are seeing an incredible amount of innovation in this space with heart, ear, and even brain-pattern biometrics at various stages of maturity and deployment.”
With the increased efficiency and affordability of physiological sensors, the possibilities do seem truly endless. In one whimsical application, Nikon’s heartography project enabled a dog to be a photographer. The artsy canine wore a Bluetooth heart-rate monitor and lightweight camera, which fired anytime the dog’s heart rate spiked. According to the photo series, particularly exciting moments included spotting other dogs, food, and random plants. The future of tracking, it seems, will even involve the family dog.