Out of Touch

Out of Touch 618 372 IEEE Pulse
Author(s): Gari Clifford, Elnaz Gederi

The impact of poor and disrupted sleep on an individual is significant, affecting the quality of life physiologically, psychologically, and financially. It is estimated that a large population of people who suffer from sleep disorders is unaware of the condition and remains undiagnosed [1], creating a need (and desire) to self-monitor. However, sleep screening is generally cumbersome and complex, requiring multiple wearable sensors (and associated wires) and experts to interpret the large volumes of data.
Recently, with the rise of the “quantified self” movement and the enormous proliferation of smartphones, more attention has been focused on consumer-level (and more affordable) wearable devices that monitor sleep, like the Polar Loop, the Nike Fuelband, and the Fitbit Flex. Even so, all wearable devices suffer from two key problems. The first issue is compliance, since the user is required to remember to charge the device, wear it overnight, and then upload the recorded data to another location (unless the device has enough energy to transmit the data over a wireless network connection). Additionally, the device can often detach from the body, run out of power, or receive incorrect input from the user. The second issue is that of localization. Since the user is required to wear the device on a specific part of the body, the sensor is often biased toward the part of the body to which it is attached. For example, a wrist-worn motion sensor will not pick up abnormal chest and leg movements, which are essential to many diagnoses of sleep disorders.
A possible solution to these issues is the use of noncontact sensors, which can be permanently attached to electrical supplies and networks and can monitor the entire body. For example, for measuring sleep activity, instead of wearing an actigraph on the wrist, an accelerometer can be attached to the bed. Walsh et al. [2] used a noncontact grid of 24 fiberoptic-based pressure sensors under a foam mattress to monitor sleep activity, restlessness in sleep, and sleeping patterns. In preliminary tests on a handful of patients, the system was shown to outperform wrist actigraphy and passive infrared motion detectors. However, the small number of subjects tested (only four) means any results should be interpreted with caution.
Audio recording is another inexpensive (and well-explored) method for monitoring sleep that does not disturb the natural sleep environment, as the microphone does not need to be placed on the subject. It is also a promising diagnostic modality. Audio recordings of sleep are used to differentiate obstructive events from snoring or normal breathing while sleeping. Due to the physiological similarities between speech and snoring, different research groups have used techniques that are often used for speech analysis to detect snoring during sleep. Several research groups [1] have analyzed the spectral and temporal structure of snoring sounds, reporting accuracies in the low 80% range for identifying obstructive sleep apnea (OSA). Recently, our group reported an entropy-based approach to assessing the regularity of patterns in audio recording between OSA and non-OSA patients. A sensitivity of 90.5% and positive predictivity of 83.5% on out-of-sample test data were achieved [3].
The electroencephalogram (EEG) is considered the gold standard for evaluating the depth of sleep and diagnosing sleep-related health issues. However, a noncontact EEG is only possible if the electrodes are fixed to a cap on the head, and, therefore, does not qualify as practically noncontact. The electrocardiogram (ECG) can also be obtained using capacitively coupled electrode technology (measured through a capacitor formed between the electrode and the electrical signal source). In contrast to the EEG, the ECG can be measured at a significant distance from the body (such as through the bed). This offers numerous advantages, since noncontact electrodes require no preparation, are insensitive to skin conditions, and can be used while wearing comfortable layers of clothing.
Using ECG recordings, a number of physiological parameters such as heart rate and respiration rate can be monitored to detect any abnormalities during sleep. Additionally, ECG recordings have been used to estimate sleep stages and analyze subjects’ sleep cycles. Despite the fact that traditional ECG recordings have been used for the analysis of sleep for several years, the use of noncontact ECGs for sleep screening has only been mentioned a few times in the literature. Lim et al. [4] proposed an indirect contact ECG system for long-term monitoring of patients’ sleep using an array of high-input-impedance active electrodes fixed on the mattress and an indirect-skin-contact ground made of a large conductive textile sheet. Their system had lower signal quality and higher motion artifacts compared to traditional ECGs, but the recordings could be used for heart rate variability analysis and detection of R-peaks. Although it should be possible to analyze sleep from this signal, the authors did not demonstrate such capabilities.

FIGURE 1 A frame from an overnight infrared video recording of an infant using a low-cost commercial Webcam with Wi-Fi streaming capability and built-in infrared illumination. (Photo courtesy of Gari D. Clifford.)
FIGURE 1 A frame from an overnight infrared video recording of an infant using a low-cost commercial Webcam with Wi-Fi streaming capability and built-in infrared illumination. (Photo courtesy of Gari D. Clifford.)

Video is clearly noncontact, yet it is perhaps the most promising modality for automatically analyzing sleep. Video recordings have been widely available in sleep laboratories for the observation of sleep behavior, body posture, and assessment of whether an apnoeic event was real. Although the automated analysis of video during sleep may seem natural as a consequence, it has rarely been used for even detecting body position and respiratory movements. There is, however, a growing (yet small) body of work that shows that the monitoring of heart rate and respiratory movements using a standard video camera (or Webcam) is possible (Figure 1). Recent work has even shown the possibility of measuring relative changes in oxygen saturation. However, there is as yet no evidence that these techniques can be used to provide useful diagnostic information from these noncontact-derived signals. One caveat is that a subject is often under bed covers, and so much of the body can be obscured so that extremely high resolution and/or motion tracking is needed to isolate the relevant body parts. Moreover, the recording environment exhibits low light levels in general, and, therefore, infrared-sensitive cameras are usually employed together with patterned bed sheets to enhance the detection of movement (Figure 2).

FIGURE 2 A frame from a video monitor on a standard commercial sleep monitoring system. (Photo courtesy of Gari D. Clifford.)
FIGURE 2 A frame from a video monitor on a standard commercial sleep monitoring system. (Photo courtesy of Gari D. Clifford.)

Video signal processing techniques (such as Eulerian magnification [5]) for the enhancement of subtle changes have also been used, but the techniques have yet to be applied to clinical data. In a recent publication [6], we have shown that it is possible to estimate the severity of obstructive sleep apnea by taking a “global” view of the video activity. By analyzing the regularity of changes in frame-to-frame pixel intensity from a standard video camera (with infrared illumination) over multiple scales, we were able to achieve 90% accuracy in differentiating obstructive sleep apneics from normal subjects.
In the smartphone app market, several applications purport to measure oscillations in the depth of sleep using the phone’s accelerometer. The user is asked to put the phone on the bed (under the pillow usually), and a pseudohypnogram (a chart of wakefulness and light and deep sleep over the night), or at least a proxy of this, is provided to the user. Often, basic statistics on how long the user spent in each category and some targets are provided. However, most evidence in the literature indicates that accelerometry is insufficient for diagnosing any sleep anomalies. It is no surprise, then, that many, if not all, of the new wave of devices and apps have little or no scientific basis. Behar et al. [7] reviewed more than 40 of the sleep apps (from both the Android market and Apple App Store) that make use of the smartphones’ on-board sensors and concluded that none of these applications were based on clear scientific evidence (with the exception of apps that implemented simple sleep questionnaires).
However, smartphones do offer great computational power and connectivity options and are particularly good at audio processing. Audio can be recorded using the internal microphone of a mobile phone, which many applications do, or using an external microphone placed either on- or off-body. Finally, with the improvements in the quality of the video cameras embedded in mobile phones, video data may be used for the analysis of sleep–wake patterns, monitoring breathing rhythm, detecting sleep posture, and diagnosis of sleep disorders. Recently, our group developed a smartphone application for the screening of sleep apnea, which was validated against clinical data [3]. A maximum accuracy of 92.3% was achieved using audio, actigraphy, and oxygen desaturation index together.
Of course, noncontact sensors also have limitations. They are often confused by multiple subjects within the range of the sensor (if the subject does not sleep alone). The sensors are often prone to much higher levels of noise and ambient interference (rapid changes in light levels, external noises, etc.). However, the ease with which one can unobtrusively place a low-cost infrared camera in a room and stream the data to complex processors provides enormous opportunities in continuous noncontact sleep monitoring and diagnostics. The advances in video signal processing are enabling us to track limb movements and multiple individuals in a sequence of frames. By combining this modality with audio analysis (arguably the most effective noncontact sleep screening tool), it is likely that we can elevate the diagnostic accuracy of a range of clinical problems well beyond 90%.


  1. A. Roebuck, V. Monasterio, E. Gederi, M. Osipov, J. Behar, A. ­Malhotra, T. Penzel, and G. D. Clifford. “A review of signals used in sleep analysis,” Physiol. Measure., vol. 35, no. 1, pp. R1–57, 2014.
  2. L. Walsh, E. Moloney, and S. McLoone, “Identification of nocturnal movements during sleep using the non-contact under mattress bed sensor,” in Proc. Annu. Int. Conf. IEEE Engineering in Medicine and Biology Society (EMBC 2011), pp. 1660–1663.
  3. J. Behar, A. Roebuck, M. Shahid, J. Daly, A. Pureza, N. Palmius, J. Stradling, and G. D. Clifford, “SleepAp: An automated obstructive sleep apnoea screening application for smartphones,” IEEE J. Biomed. Health Informat., vol. PP, no. 99, pp. 1, 2014.
  4. Y. G. Lim, K. K. Kim, and K. S. Park, “ECG recording on a bed during sleep without direct skin-contact,” IEEE Trans. Biomed. Eng., vol. 54, no. 4, pp. 718–725, 2007.
  5. H.-Y. Wu, M. Rubinstein, E. Shih, J. V. Guttag, F. Durand, and W. T. Freeman, “Eulerian video magnification for revealing subtle changes in the world,” ACM Trans. Graph., vol. 31, no. 4, p. 65, 2012.
  6. E. Gederi and G. D. Clifford, “Fusion of image and signal processing for the detection of obstructive sleep apnea,” in Proc. 2012 IEEE-EMBS Int. Conf. Biomedical and Health Informatics (BHI), 2012, pp. 890–893.
  7. J. Behar, A. Roebuck, J. S. Domingos, E. Gederi, and G. D. Clifford, “A review of current sleep screening applications for smartphones,” Physiol. Measure., vol. 34, no. 7, pp. R29–46, 2013.