Link between Hearing and Bodily Sensations

1. Introduction

Humans obtain information about the world through the so-called five senses (vision, hearing, touch, taste, and smell). Therefore, understanding the characteristics and mechanisms of these senses is essential for transmitting information from one person to another adequately. The role of the senses is widely recognized to be to gain information regarding the world around us. However, simply knowing our surrounding environment is not enough for us to act within the environment. We also must know about our own body. By knowing our own body, the surrounding environment, and the relationships between them, we are able to take appropriate action within the environment. We also obtain information about own body through our senses. This implies that by managing sensory information regarding the body, it is possible to control what a person feels (creating illusions) about his or her own body. It is well known that the somatic senses and vision play important roles in the perception of the body. In our research group, however, we have been conducting research focused on hearing. This article introduces the role that hearing plays in body sensations.

2. Bodily perception and vision

Of course, information about the body is obtained from the somatic senses, which relate to the body (such as the sense of touch, proprioception, and interoception). We can perceive something touching the body through the sense of touch, and the position and motion of different parts of the body through proprioception (the sense of extension of the skeletal muscles). Interoception gives information regarding the physiological state inside the body, and our sense of temperature tells us the temperature of our body and of objects that touch the skin. However, looking at the body is also an important way to know its state. For example, it is not possible to know the length of an arm or a body part accurately through somatic sensation alone. It has been shown that visual information is important in understanding body shape. Many visual clues are also used in determining whether something is touching the body.

The rubber hand illusion is a famous example of a phenomenon that shows the importance of vision in bodily perception [1]. The right hand of a participant is hidden, and a rubber model of a right hand is placed in front of him (Fig. 1). The experimenter repeatedly strokes both the hand of the participant and the rubber hand in the same position at the same time. The participant sees the rubber hand being touched in the same way he feels his own hand being touched. After experiencing this for a few minutes, the subject starts to feel as though the sensation of being touched is coming from the rubber hand, and furthermore, that the rubber hand is part of his own body. It has been reported that if the rubber hand is then subjected to an apparent injury while the illusion is in effect, the physiological responses to the injury are similar to those when the participants’ own hand was injured [2]. When receiving both the tactile information from our own hand being touched and the visual information of the rubber hand being touched in the same way, our perceptual system interprets that the model hand is our own hand.

Fig. 1. The rubber hand illusion.

3. Touching with sound

To date, there has been very little research into the role played by the sense of hearing in how humans perceive the body. However, a sound is often produced when something touches the body or when we move our bodies, for example, the sound of footsteps when we walk. Such relationships between the body and sound suggest that the sense of hearing can contribute to bodily perception.

We conducted an experiment shown in Fig. 2(a) [3]. In the experiment, a microphone was placed in the ears of a dummy head, and the participating subjects were able to hear the sounds heard by the dummy ear. The dummy ear was then tickled using a small brush. The experiments were divided into two cases; in one case, the sound was presented very close to the subject’s ear (i.e., through headphones), and in the other, the sound was presented from a loudspeaker located at a distance of 70 cm from the ear (with sound intensity equalized according to head position). Then, after listening to the sound for 30 seconds, the subjects were asked to rate several items on a scale of 1 to 7. The results are shown in Fig. 2(b). When the sound was presented near the ear, the subjects felt as though their own ear was being tickled, but when it was presented at a distance, they did not perceive it that way. When they only viewed a video of the dummy ear being tickled but did not hear any sound, they did not feel as though they were being tickled. In the experiments in which the subjects only heard the sound and did not see the dummy being tickled, the subjects had no idea what the sound was. This shows that the subjects were not simply imagining they were being tickled when hearing the sound. When the sound was presented close to the ear, the subjects perceived that they were actually being touched, even though they were not. Various other tests were done that involved discriminating the locations and temporal order of tactile stimuli, and measuring simple response times for tactile stimuli and sounds. The results showed that when sound is presented close to the head, it affects the sense of touch in the vicinity of the sound [4]–[6].

Fig. 2. Tactile illusion induced by sound. (a) Tickling with sound, (b) results of the experiment.

The relationship between hearing and the sense of touch in the head is so strong that touch is perceived just by hearing a sound. However, it is also known that the sense of touch can change according to the sound when a sound is produced by a touch of another part of the body. For example, when the sound of rubbing both hands together is captured with a microphone and heard through headphones, if the high-frequency components are amplified, the listener’s hands tend to feel dry, but if high-frequency components are attenuated, the hands are perceived as damp and heavy [7]. Sound effects in television and movies are well known, but sound effects can also affect the sense of touch in our hands. From the time we are born, whenever something touches our face or head, or when we touch something with our hands, and a sound is produced at the same time our skin feels a sensation, the sensations are experienced together, and we learn the relationship. Because of this relationship, a sensation of touch can be produced or changed by sound, even if we are not actually being touched.

4. Lengthening of arm through sound

Next, we introduce an illusion in which the perceived shape of the body is altered by hearing sounds. What we learned from the example of the rubber hand illusion introduced in Fig. 1 was that what we recognize as our own body is constantly being updated based on sensory information from somatic, visual, and other senses. If there are inconsistencies in the sensory information input, our awareness of what our own body is can change very easily. Analyzing illusions related to the body such as the rubber hand illusion reveals how we perceive our bodies.

Traditionally, the somatic senses such as touch and proprioception, as well as vision have been understood to be important in recognition of the body. However, we have recently discovered that hearing also contributes to recognition of the body [8]. We created conditions in which the sound of tapping on the floor is heard from a farther distance than where the tapping is actually occurring, as shown in Fig. 3. In the experiments, subjects were blindfolded and asked to kneel on the floor. An object was used to lightly touch the subjects’ right forearms and left upper arms at two points ranging from 2–6 cm apart. This tactile stimulation was carried out both before and after the subjects listened to a sound coming from speakers placed on the floor near them. As they listened, they were asked to tap the floor with their right arm. The timing of the sound and the position of the speaker were varied, with the speakers getting further and further away. The illusion occurred when the sound was produced at twice the distance to the location where the subjects tapped. When the subjects’ arms were then stimulated after they had tapped the floor, they tended to say that the distance of the two stimulation points was farther apart on their right arm than on their left arm. This creates a spatial inconsistency between the somatic senses and hearing. When experiencing these conditions, our perceptual system may be interpreting them as, the sound is coming from where my right hand is tapping, and my right hand is tapping far away. Further, if that is the case, we might also feel that my right arm must be pretty long.

Fig. 3. Action sounds heard from a distance induce the illusion of a longer arm.

Experiments conducted to investigate this showed that, in fact, subjects perceived their right arm to be longer than it actually was. After experiencing this condition, objects presented on the right arm were felt to be longer than those felt before the experience. The fact that the same objects that touched the skin of the arm were perceived to be longer after the experience suggests that the arm is recognized as being longer in the brain. The illusion occurred when the sound was produced at twice the distance to the location of the tapping, but not at four times the distance. It was also important that the participants actively tapped the floor themselves. Additionally, it was shown that the illusion occurs at an unconscious level. These findings show that the spatial position of sounds produced by our own bodily motions provide unconscious clues in how we perceive our own bodies.

Our recognition of our own bodies is constantly and flexibly being updated. This suggests that any bodily sensation can be produced by controlling it appropriately. Further, it should be possible to make the experience of virtual reality spaces more realistic by inducing such bodily sensations. These changes in body recognition are also thought to be related to how users perceive a tool as becoming part of their body when they become accustomed to using it. The extent to which bodily perception can be extended, spatially and temporally, is a topic we are considering for future research.

5. Future developments

We believe that bodily sensations are deeply related to the reality of experiences. Most experiences that involve no bodily sensations feel unsatisfying in some way. Most seeing or hearing sensations relate to phenomena that are distant from the body, but touch is said to be able to confirm the existence of what is being touched [9]. Further, what we recognize to be our own body is what we recognize as ourselves, which is the basis for having a concept of the self. A concept of the self is also necessary for understanding others, so bodily sensation is fundamentally important for our communication.

The fact that the sense of hearing contributes to perception of the body is extremely important for information and communications technology, which is a medium for communication. It is difficult to transmit bodily sensations such as touch, both technically and in terms of cost, but it is relatively easy to transmit sound. It seems that by arranging how sound is presented, it should be possible to convey experiences with a level of reality that accompanies bodily sensations. Elucidating the mechanisms of body awareness should contribute to realizing better and deeper means of communication in the future.

References

Norimichi Kitagawa Senior Research Scientist, Human Information Processing Laboratory, NTT Communication Science Laboratories. He received the Ph.D. degree from Tokyo Metropolitan University in 2003. He received a JSPS (Japan Society for the Promotion of Science) Research Fellowship for Young Scientists (Superlative Postdocs) and spent time at the University of Oxford, UK, in 2003. Upon his return, he took up a lectureship at Kanazawa Institute of Technology. He joined NTT Communication Science Laboratories in 2005. He is interested in how we perceive the world around us and how our perceptual systems process the information from different senses such as hearing, touch, and vision.