Man-machine interfacing is one of the most interesting areas of research in biomedical engineering. The aim of such interfaces is to collect information from and present information to the human body in order to enable and streamline interactive processes involving human users. But one artist/researcher and technology entrepreneur, Marco Donnarumma, has taken this concept one step further. Through his project, XTH Sense, he had the vision of treating the human body as a complete instrument in itself. In 2012 his novel idea was named the “world’s most innovative musical instrument,” winning the first prize of a renowned competition organized by Georgia Tech Center for Music Technology, Atlanta, Georgia. Pulse Deputy Editor-in-Chief, Ahmed Morsy, had the chance to speak with Marco Donnarumma about his unique and novel ideas.
Morsy: How did it all start? And what is the vision you wanted to advance?
Donnarumma: The story of the XTH Sense™ began in 2010, when, as a touring performer and musician in search for new ideas and work methods, I enrolled in the Sound Design Master by Research at Edinburgh University. The initial idea was to create an instrument that would emphasize the bodily expressivity of a player on stage and, at the same time, help the player focus on subtle nuances of her physical movements. In other words, an instrument that could be configured with the human body. It had to be something other than a hand-held controller or a camera-based one, which are the most common types of musical human-computer interfaces.
These interfaces did not meet my needs for two reasons. First, a controller in a performer’s hands tends to attract the attention of both audience and player. One is more interested in grasping the interactive mechanism of the controller rather than the physical nuances of the player’s performance. Second, camera-based interfaces are bound to channel the performer’s attention on her position in respect to the camera itself, rather than make her more attentive on the nuances of her movements. What was needed was an instrument that was almost invisible and that could access the inner physical activity of a performer’s body in the form of physiological processes.
Based on these criteria, I began researching physiological computing and, in the beginning of 2011, I created the first XTH Sense using a custom mechanomyographic (MMG) sensor and a software framework developed in Pure Data (Pd), a graphical programming language. The first version of the XTH Sense was simple but efficient, and in a couple of months I composed and performed the first piece for the XTH Sense entitled “Music for Flesh II,” which I still perform today. In that piece, the instrument amplifies the MMG signal, or muscle sounds, produced by physical arm gestures and allows me to live sample those same sounds by exerting different force levels. In other words, the player’s body produces sounds that are digitally manipulated by the software algorithms according to variations in the muscle activity.
[accordion title=”Tech Specs of XTH Sense”]
The XTH Sense combines gestural interaction, physiological monitoring, and interactive music into one creative instrument. What distinguishes the XTH Sense from the other wearable devices on the market is its capacity to amplify otherwise inaudible bodily sounds, measured as mechanomyogram (MMG) and other bioacoustic signals, and transform them in interactive music. Since it is plug & play—any kind of movement immediately results in an audible sound—as well as highly-sensitive (it captures a subtle tremor of a finger even when worn on the arm) it can be used by everybody, independent from their age, body size, or health condition. This means that differently abled people, or those who suffer from a muscular condition, can use it too, with their own gestures.
The XTH Sense is a tiny, sleek, and ultra-light 2-inch wearable. It weights 14g and it comes with a soft-touch material. Using differently sized bands, the device can be worn on any part of one’s arms, legs, and torso. The device runs on 1.8-3.3V provided by a rechargeable LiPo battery (400mAh), which lasts for about 15 consecutive hours. The device also includes a LED that can be synchronized with any of the biosignals or motion data captured in real time. The XTH Sense provides multiple sensing modalities: these include bioacoustic and temperature sensors, along with a 9-degrees-of-freedom IMU, embedded with an accelerometer, a gyroscope, and magnetometer. The bioacoustic sensor captures any sounds produced within the user’s body, and preset algorithms are used to emphasize the sound of the heart beating, the blood flowing, or the muscles contracting.
The sound and data generated from the user’s body are sent in real time over a wireless connection (+-15m distance) to a receiver connected to a laptop or smartphone. Here the raw sound/data and the related features can be accessed through our API in three ways: through a dedicated XTH app; through the XTH plugins running in Python, Max/MSP, PureData, AbletonLive, Reaktor, Unity; or through any MIDI- and OSC-enabled software. The XTH algorithms analyze the acoustic properties of the bodily sounds and specific aspects of the motion data, and then associate that information to musical parameters. We use a custom set of feature extraction algorithms we have developed in the past years; these draw on the biomedical literature to identify high-level aspects of gestural expressivity and stress levels.
In more detail, the XTH Sense uses an Atmel ATmega256RFR2, a 16 Mhz MCU, a 8k EEPROM, a built-in radio transceiver (2.4GHz using 802.14.4) and firmware/software sampling rate is 500Hz. Two wearables can be connected to one single receiver, making it easier to explore biosignals and motion data on different parts of the body at once. Our API is written in C, and the firmware can be re-configured on the fly using either a GUI or the terminal.
[/accordion]
Morsy: You managed to build a community around this project. How were you able to do that, and is there a particular profile for members of such a community?
Donnarumma: Indeed, this is the part of the project that I find more rewarding. There exists a varied and steadily growing community of artists, researchers and hackers that have used, or still use, the XTH Sense for a broad range of applications. There is not a typical XTH user; if one’s area of interest touches upon either sound, music, interactivity, physiology, artistic performance or any combination thereof, then the XTH Sense is likely an instrument to have at hand. Projects so far have ranged from a large theater production by Damir Martinovic, a seminal Croatian rock musician; a master’s thesis on neuromuscular rehabilitation by a student in India; a virtual reality game using the XTH Sense with the Oculus Rift created by a group of independent developers; to courses on music technology and DIY tech and more.
The growth of our community is possible for two interrelated reasons. First, the XTH Sense is completely open source: while we own both a trademark on the XTH Sense brand and the copyright of the XTH wearable hardware, software, and product design, we have applied GPL and CC licenses to guarantee the right of our users to open, hack, and customize their XTH Sense. Anybody who redistributes a modified version of the XTH Sense must use the exact open licenses we use (and cannot use our brand, unless otherwise agreed). This is crucial in order to develop a community able to foster different kinds of creativity and, by doing so, create and share knowledge.
Second, because the project is completely open, I have been able to run workshops and master classes at universities, festivals, and media labs worldwide, where the participants could build from scratch their own XTH Sense biowearable and directly experiment with the technology at a very low cost (the first version of the XTH Sense DIY kit cost about £20). Between 2012 and 2014 I have taught over 35 workshops where participants learned from my expertise, and I learned from their feedback. I was lucky enough to accumulate a great deal of knowledge by interacting hands-on with people around the world and this has continuously informed the design of the XTH Sense, while expanding our community.
Building a community is about encouraging users’ empowerment through a particular set of tools and a specific, and yet open mindset. Often empowerment is confused with usability. But making a device user-friendly is not equal to empowering its users. Empowerment means to acquire specialized knowledge by understanding and reproducing a process (or a technology). Empowerment also implies the cultivation of critical skills that can lead to the conception of truly novel and original paradigms. In this sense, the whole history of knowledge can be thought of as a jigsaw puzzle. Each generation adds puzzle pieces which have only a small part of the picture on it, and gradually the bigger picture is unfolded.
Morsy: How is it possible to monetize your ideas?
Donnarumma: Very good question. The premise is that, for several years, the thought of monetizing the project has not interested me. As long as XTH was a personal project, I found it rewarding enough to see how useful the XTH Sense is to other people. Today, XTH is a company—co-founded with artist and creative technologist Heidi Boisver—involving about ten core collaborators, partners, and various consultants; therefore the project must be monetized in order to keep doing research and providing the best service to our community. Our strength lies in a particular approach to the wearable market, which combines our passion for openness and an unconventional approach to the wearable market.
Despite the accessibility that companies seek by providing an off-the-shelf, creative product, be it software (SW) or hardware (HW), its source or schematic is often protected by copyright laws that impede its open re-distribution or modification. A user does not buy the product itself, but rather a license to use it; in other words, the product is simply and effectively closed. In contrast, DIY and Free Libre Open Source Software (FLOSS) strategies deliver open SW and HW that can be freely re-distributed and expanded by the community. Nonetheless, many times DIY projects are not concretely accessible by a wide audience, as specialized background knowledge is needed. So we face a paradoxical model.
Companies have grasped the production potential of the DIY community, and have began to slightly loosen their policies, so that makers and hackers would develop new applications for their devices. The marketing model of (and the hacking hype around) the Microsoft Kinect and the Nintendo Wii exemplifies this strategy quite well. At the same time, many emerging technologies in the fields of virtual reality, personal, and physiological computing are still almost completely closed, and either do not provide access to raw data, or charge data-access fees.
What is our stand in this scenario? Our approach is simple: we offer a reliable and highly efficient biowearable technology at a competitive cost and with open source licenses. Users have access to both raw data and the feature extraction methods we have developed. They can extend our algorithms and adapt them to their needs. But, our marketing strategy has a further added value. Differently from many wearable startups who focus primarily on functional applications of bodily data, like sleep and activity trackers or physiological monitoring, our product is about enabling new modes of expression through sound, the human body, and physiology. In my view, technology does not only do things for us, it does things with and to us. And the XTH Sense is about combining technology and the human body to enable novel modes of expression and shared experiences. Our bodies and technologies are not only data-mining means, they are highly expressive media.
Morsy: Are you currently working on any new projects?
Donnarumma: Well, truth to be said, this period has been positively intense. The most important project at the moment is the launch of the XTH Sense on the market through a series of press releases, videos, and a crowdfunding campaign. The aim of the campaign is twofold: to diversify our community, directly involving a larger number of researchers, educators, and creators (in music, dance and hacking); and to raise the funds needed to begin a large-scale manufacturing of the instrument, and thus make it more immediately available worldwide. We are also applying to accelerators and accepting seed investment proposals.
What is more exciting is that during the crowdfunding campaign we will learn more about the needs of our community and how we can respond to them in the best way. A critical next step for the product is the possibility to validate its usefulness for rehabilitation. To that end, we are working on the possibility of running a series of user studies in collaboration with local institutions and researchers.
On the other hand, I have recently submitted my PhD at Goldsmiths, University of London, and moved from London to Berlin. There, I will continue my scholarly research on technological embodiment, physiological computing, and musical performance. At XTH we will be focusing on further research and development of both the XTH Sense and new product concepts. On the horizon are a few publications, most notably a sound and video anthology on the theme of biophysical music that will be published in the Winter issue of the Computer Music Journal (MIT Press), and a book chapter on the principles and challenges of physiological computing in musical performance, with a focus on muscle sensing, to be published by Springer next year. Well, and I should not forget the publication of my PhD thesis next year!
Morsy: What is your advice to anyone who wants to work at the intersection between arts and science?
Donnarumma: The field of art and science is still young and yet it evolves rapidly. We do not have consistent criteria to evaluate the work that is being done in this field, and too often, projects that lack scientific or artistic rigor are assessed on the basis of the beauty they display. This is unnecessary and counterproductive, for it encourages a superficial and unsophisticated approach to the making of art through science—an approach that often overlooks both considering relevant findings and acknowledging the work done by others in the past in favor of an “awe effect.”
So my advice to an aspiring artist or scientists wanting to work in this field is to be rigorous, to strive for sophistication of thought, and to never forget that creativity is not assessed in terms of displayed beauty. Rather, beauty is something subjective that emerges from the process of creation and the messages being conveyed by an artwork; something which at first seems an unfamiliar or disturbing piece of art may eventually reveal itself as a most graceful experience.
Finally, and perhaps most importantly, it is fundamental for artists and scientists working in this field to be aware of their cultural and political responsibility. The way an artist use a particular technology or a particular human body in a public exhibition or performance has a political value, whether or not the artist chooses to make full use of it. It has a political value because it reinforces, alters or disrupts the societal criteria against which both technologies and human bodies are assessed.