The objective tools of engineering should be brought to the challenge of understanding emotions in people, and this could change the way we approach many health conditions, including epilepsy and depression, according to Rosalind Picard, Sc.D., a professor in the Media Laboratory and the director of affective computing research at the Massachusetts Institute of Technology (MIT). She coined the term affective computing and defined it as computing that relates to, arises from or deliberately influences emotions. She is also a founding member of the first IEEE Technical Committee on Wearable Information Technology, which was the precursor to the quantified-self movement that promotes self-monitoring using wearable sensors and other technologies.
IEEE Pulse recently spoke with Picard about her work evaluating emotions, including her role in developing a watch to detect epileptic seizures, and her conviction that wearable sensors will soon be able to help identify early signs of depression and quite possibly prevent it.
IEEE Pulse: What interested you in this field?
Rosalind Picard: I began thinking about whether there was something measurable in the body that changes with emotion. I’m talking about real physiology and biochemistry: Does your heart beat differently with different emotions? Do your muscles tense differently? Do your pupils dilate differently? I am a machine-learning/signal-processing person, and to me, these are all signals that, if I could quantify them somehow, then I could model them and use machine-learning to see if there were any trends in the data that were predictive of happiness, sadness, stress, or other states. Lo and behold, there were.
IEEE Pulse: What measurements did you focus on?
Picard: We started to measure just about everything that people were comfortable with letting us measure. We’d put muscle-tension sensors on the face and the arms; we’d put electrodes on the scalp, the face, and the skin on the hands and feet; and we’d measure electrocardiogram, electrodermal activity, electroencephalogram (EEG), electromyogram, and respiration. We also combine this with other information about context.
IEEE Pulse: How did you translate these data into something related to emotion?
Picard: One of the challenges is determining the true label of emotion. Is it what people tell you? Psychologists will say that the true emotions are what people tell you they’re feeling. And there are times when that is a pretty good label. However, we also knew that not everyone experiences emotions the same way. Some people are not aware of their changing emotions and have difficulty reporting them until the changes become very extreme. And some people can’t use language or are too young or injured to use language. So we realized that people’s self-reports are not always going to be reliable.
To get around that limit, we put people into situations where we would be very likely to elicit certain feelings (and monitored signals related to those feelings). For example, we gave a group of people from MIT a novel computer game and told them we wanted their feedback on the game. We said they would get one of two US$100 prizes if they were a top scorer, and since many MIT people are not money motivated, we said that their score on this type of game was a function of their intellect and that we were also giving the game to people from (rival) Harvard University. While they were working really hard to score in the game, we made things go wrong, such as the timer clock would go forward but their score would not. We elicited frustration. Even if a person doesn’t report their frustration levels climbing steadily, we can see changes that are measurable: sometimes their skin conductance, muscle tension, and heart rate go up; sometimes they squeeze the mouse harder or repeatedly pound on the input device. The circumstances help us label the emotion. Furthermore, we could measure and quantify those signals (e.g., noting upward trends in heart rate or muscle tension), and the computer is taught to recognize them.
IEEE Pulse: How can you apply this to a medical condition?
Picard: That gets a bit trickier, and it depends on the condition. We’ve done a lot of work measuring stress in people, particularly examining skin conductance at the wrists (signals increased with mild autonomic stressors and decreased with relaxation). While we were getting data from a little boy, who was the younger brother of one of my students, we saw a whopper of a stress response, but it was on just one side of his body. As an engineer, I thought the sensor must have just not been working right. How can a person be stressed on his left side and not on his right?
After unsuccessfully debugging it, I gave up and called the student at home to see if he had any idea what happened to his little brother that might cause these completely inexplicable readings. I gave him the exact time and date that this happened, and he said it was right before his brother had a grand mal seizure.
IEEE Pulse: So the brother had epilepsy, but you didn’t know it beforehand.
Picard: Right. I started looking into it and called another student’s dad, brain surgeon and epilepsy specialist Dr. Joseph Madsen at Children’s Hospital in Boston. I asked him if it was possible that there could be a huge sympathetic nervous system surge 20 minutes before a seizure. He said that some patients had their hair stand on end before a seizure and that it would happen on just one arm. I explained our data, and the next thing I knew, we were adding our wristbands to an approved study of 90 hospitalized children to see if others would have these responses on their wrists. Seizures are like little electrical firestorms in the brain, which can spread, just like a little brush fire can spread and take over the whole forest. Usually, you have to have an EEG to measure them. Could we really “see” them on the wrist?
One of the surprises I learned was that brain activity deep in the brain doesn’t show up directly on the scalp electrodes of an EEG; the EEG picks up electrical activity in the cortex, closer to the scalp. However, some deeper brain regions, when activated, produce signals that we can measure on the wrist. From that, we learned that our wearable sensors could pick up patterns that could be potentially lifesaving for certain epileptic events. Usually, the response is picked up after the seizure starts but before the time where a seizure might potentially become life-threatening.
IEEE Pulse: This motivated the development of the Embrace watch through Empatica, Inc., of which you are a cofounder. Is that correct?
Picard: Yes. I’m wearing a working prototype right now, and we plan to ship it to our first customers in April 2016.
IEEE Pulse: What physiological traits is the watch picking up?
Picard: The two main signals it’s measuring with high quality are skin conductance and several types of motion.
IEEE Pulse: Will this technology have application for other conditions?
Picard: Yes, we are beginning medical studies right now to quantify autonomic and other objective responses that are hypothesized to be potential biomarkers. We’re currently in studies of depression, posttraumatic stress disorder, and migraine; and we’ve also participated in other studies such as stress-related overeating, behavioral disorders where children act out with bad behaviors, characterizing states that might lead to meltdowns in autism, and with other areas that assess human stress and experience.
IEEE Pulse: Describe your work with depression and what you hope to accomplish there.
Picard: Currently, a person goes in to see a doctor when they already need help: the way the system works now is largely based on assuming everything is fine with you until, one day, you get a diagnosis and you’re sick. Then they usually give you pills to attempt to fix you, which often don’t work. Wearables are going to change the way this system works. Wearables collect data that can be mined to find behaviors—internal and external—that relate to resilience or vulnerability, or the transition from the former to the latter. Hopefully you’re strong and resilient right now. However, what if six weeks from now, your sleep is off, your stress exhibits highly atypical and imbalanced patterns, and you’ve significantly reduced your physical and social activity? What if we can show people that certain daily choices they make increase or decrease their risk of becoming depressed? I believe that from a lot of data, the system can learn how to forecast which measures make depression more likely to happen. We are aiming for a kind of emotional weather forecast—not perfect, but it may suggest you make preparations for a possible storm. Even better than issuing a warning of bad weather, this new research may give you the chance to change the weather—to change the health outcome. I think we may be able to prevent a lot of depression by observing behavioral changes in an individual patient early and then nudging them in ways that can head off disease. Perhaps you are at risk of depression (many college students in intense programs are) and your behavior is changing to look increasingly like those who have become depressed. Wouldn’t you like an early warning, before it becomes incapacitating, and get help that keeps you well?
IEEE Pulse: So what is the focus of your work right now in this regard?
Picard: We want to be able to understand what these signals look like as the patient is transitioning to disease, develop algorithms that can pick up the warning signs, and provide enjoyable ways to engage patients in prevention. This would be much better than waiting until they’re in trouble and have to go to the doctor because it has progressed to a serious medical condition. A lot of money has been spent showing that proper attention to diet, exercise, and other behaviors can significantly reduce cancer risk; it is time to do the work to identify behaviors for reducing the risk of mental health disorders. With tools that help measure emotional data, we are better equipped than ever to start this challenge.
I have worked with brilliant psychiatrists, neuroscientists, and psychologists who say they don’t know how to solve this problem. It’s time for engineers to bring quantified-self measurements and the new ability to learn from rich, continuous, real-life data, showing what’s changing in our behaviors and physiology over weeks and months, to help understand the precursors to depression and stress-anxiety disorders.
My goal is to do the new science that can prevent 80% of mood-related disorders. Why can’t we do that? Why not try?
