A Scientist who Studies Human Beings from an Artistic and Philosophical Perspective
The ability to convey human sensory information via vibrations on a smartphone—a device now used by about 65% of the Japanese population—would no doubt lead to a whole new style of communication. Distinguished Researcher Junji Watanabe of NTT Communication Science Laboratories is researching the relationship between the human senses and the environment. We asked him about the essence of his current research and his approach as a researcher.
Keywords: subconscious, haptic information, onomatopoeia
Standing in the middle of science, art, and philosophy
—Dr. Watanabe, please tell us about your recent research activities.
In person-to-person communication, the information exchanged by words is no more than a small part of what each person thinks about in his or her mind. When I encounter another person and engage in a conversation, I give more importance to things that are not expressed in words or things that I am not conscious of, even though I may have subconsciously felt them. This is what I believe to be the meaning of directly encountering and talking with another person, and I am interested in the moment that such things come out.
This kind of relationship between the conscious and subconscious also holds for things that move one’s body. In everyday life, when we consciously set out to move our own body, there are actually more complex movements taking place on a subconscious level. Even for movements that we may think to be simple, there are many subconscious movements in the background. With this in mind, I recently broke down movements that are usually performed subconsciously into detailed movements and sensations and created a video of the results like a musical score. This video was exhibited under the title “A Dialogue with the Subconscious, the Most Familiar Other” at the exhibition “ATHLETE” held by 21_21 DESIGN SIGHT in Roppongi, Tokyo from February–June 2017 (Fig. 1).
Athletes are known to practice while paying detailed attention to each and every bodily movement and to then train themselves until those movements become subconscious actions. They feel that such a disciplined approach will enable them to excel at the time of a match without having to think about it. Talking about this with the exhibition director led to this work of mine. Thinking that not just athletes but ordinary people too should have access to such a process, I decided to make conscious in an easy-to-understand form those things that people are not usually aware of. Specifically, I took up actions that require some practice in our daily lives such as typing or doing finger motions on a smartphone screen (tapping, swiping, etc.), and I analyzed them from the viewpoints of muscle movement in the bending and extending of fingers and sensations such as fingertip pressure and position. I then represented the results of this analysis like a musical score.
—It’s surprising that presenting such an exhibition would be included in a researcher’s activities; you’re acting more like an artist.
I am interested in human beings and how they are changed through technology. I think that it’s important to understand the essence of human beings from a scientific viewpoint. However, in creating works, my aim is to convey that understanding as a personal experience much as an artist would do, so when viewed from the outside, I may appear to be a mix of a scientist and an artist.
In any field, there is context and format, and to date, I have mainly been involved in writing journal papers, the main format of science. However, I have also participated in exhibitions, one of the main formats in the world of art. In that case, though, it’s not that I would exhibit a “work” within the context of art. Rather, I would focus on making experiential inquiries into the meaning of human beings and technology by questioning the meaning of various things, as in “Can’t this also be interpreted in this way?” or “Is there anything else that has this meaning?” Of course, activities such as these have some relation to art and philosophy.
“Now” is the result of trial and error—retracing your footsteps to find your objective
—What motivated you to use this approach in representing your research?
From 2005 to 2009, I served as a PRESTO (Precursory Research for Embryonic Science and Technology) researcher at the Japan Science & Technology Agency (JST), where I conducted research and exhibited that research under the theme of “Fundamental Science and Technology for New Tactile/Haptic Expressions.” During my time as a PRESTO researcher, I showcased my work every month, if not every week, and I can say that my present way of thinking grew out of a trial and error process at that time.
Something I try to be careful about when demonstrating or exhibiting my research is to refrain as much as possible from limiting attendees in the way they view that research in the sense of “It must be thought of in this way.” Of course, I give some explanations on what I am exhibiting, but regardless of how it is interpreted, that interpretation is what constitutes truth for any one attendee, so I always hope that a dialogue about interpretation occurs as a result. The type of medium used to present one’s research—journal paper or exhibit—depends on the content of that research, but the way of thinking behind that research does not change. I choose an optimal method of output for each case.
In fact, a variety of formats can be used to output the contents of one’s research, and my use of several types of formats is perhaps unique. At a design or art exhibit, I try to present my research in a way that can be experienced without an explanation, and in research demonstrations such as at the NTT R&D Forum, I tend to explain my research in a step-by-step manner. Another approach I might employ is to have recipients experience the results of my research in a sequential manner over a somewhat long period of time, as in a workshop, or I might convey my research theme through the medium of a magazine. Furthermore, in addition to such momentary experiences, I strive for exhibitions in which recipients can take something back with them—that is, something they can keep in mind long after the exhibit is over.
—You have been involved in diverse research fields. Can you tell us something about your research history up to the present?
I am currently involved in a variety of endeavors including research on the human sensory mechanism, haptic design, and creation of wellbeing guidelines. This may seem curious from the outside, and people may ask: “Why are you involved in so many disconnected things?” But for me, these are all connected.
In the third year of my university studies, I had to choose a field to specialize in. My first choice at that time was virtual reality, my second choice was cognitive science, and my third choice was philosophy. Consequently, I joined the virtual reality laboratory and researched human perception, and it was not long before I developed an interest in the relationship between the human senses and technology.
Moreover, looking back at my childhood, I remember being more interested in principles than material phenomena. For example, I would be more interested in “Why is this the case?” or “How was this derived?” than coming up with conclusions or answers in a test. Studying by rote memorization even in fields such as physics and chemistry was not very appealing to me; I was more interested in the underlying rules. Likewise, with respect to human beings, I was not very interested in superficial speech or behavior. What interested me were the principles of sensation and movement giving rise to such human behavior and the principles of the human mind explaining why it occurred.
I believe that researching the human sensory mechanism is a primary way to approach the principles of the human mind, and that the sense of touch can act on emotions and the subconscious. One example is the “Heartbeat Picnic” workshop that enables participants to experience what it would be like to hold their own hearts in their hands. In this workshop, when you hold a stethoscope against your chest, it causes a cuboid-shaped box to vibrate in sync with your heartbeat, thereby producing a feeling through the sense of touch that you are really alive (Fig. 2). I have also conducted a workshop that attempts to visualize the perceptual space of tactile texture through the use of Japanese onomatopoeia (Fig. 3). The idea here is to draw a map of tactile sensations that are attuned by using onomatopoeia, which are words that mimic the sound of various objects, or words having a more delicate and intuitive nature rather than simple adjectives such as “rough” and “hard.” In addition, I recently began research in the field of wellbeing as the ultimate motivation behind human behavior.
Searching for the essence of all kinds of things and the relationships between them
—What, then, is the common characteristic of these diverse research fields?
I would say that my personal research theme is to examine how the senses of individual human beings create communication and how society came into existence. I also want to search for the essence of these developments and the relationships between them.
For example, onomatopoeia refers to special words that make a connection with the senses by relating sound and meaning. It’s easy to relate the ‘s’ sound in a Japanese word like “sara sara” with a smooth sensation. In contrast, the word “zara zara” vocalized by vibrating the vocal chords with the same mouth position as ‘z’ relates to a rough sensation. This is probably because air vibrates more vigorously at the gums when we make ‘z’ sounds. Through the research on the connection between sensations and words in this way, the “image of a human” in my mind gradually began to utter words.
In addition, while I had been observing humans and researching what they were feeling and thinking at certain moments in time, I am now beginning to view them from a continuous perspective. For example, when a person is motivated to behave in a certain way, how then does that person change the way of dealing with whatever he or she encounters? This constitutes a system loop within the human mind, which is something that I would like to pursue. In this regard, the increasingly popular practice of “mindfulness” can train a person on how to focus one’s awareness on the present, that is, on what one is experiencing in the here and now. However, in addition to the immediate present, a human being features an ongoing internal loop that uses present happenings to trigger a look back at the past and maybe a look into the future. I am examining the role of the senses in this loop and searching out what might be possible in this area through technology.
For example, we touch our smartphones to get information. If haptic information should now be given in response to such an action, our feelings may intensify with respect to that information, and our motivation to do something may likewise become stronger. I believe that new technology of this type could be used to influence a person’s mental cycle, change a person’s behavior, or even improve a person’s emotional state.
—The key to finding out what people are feeling or trying to convey seems to be interaction and communication.
We cannot feel the exact same sensations as those of another person, and in terms of communication, it is impossible to “completely” understand what another person is saying. Consequently, since words uttered to someone cannot be fully understood as the speaker intended, the sensation that “something is being understood” can be just an illusion. In other words, when a verbal exchange with someone proceeds without any problems, the sensation that “we are communicating” can likewise be an illusion. Speech consists of a “what” element indicating what is to be conveyed, and a “how” element indicating how that information is to be conveyed. “What” is often expressed by language in a conscious manner, while “how” is strongly related to subconscious movements such as body and hand gestures and facial expressions. Furthermore, communication can be achieved simply on a “how” basis without understanding “what.” In fact, there are times in which “how” plays a more important role in communication. We can read a person’s intent not only from “what” but also from “how” in a way that enriches communication.
When thinking alone, one only thinks about what can be conveyed through words, but if I can get another person to pick up on what I am subconsciously thinking about, I can uncover what was ambiguous and hidden in whatever I was thinking about. At that time, such an idea is not only mine nor the other person’s—rather, it is something that arises between both of us. In communication between two people, each person is thinking about a variety of things above and beyond what each is saying. These thoughts influence the subconscious “how” portion of communication and are somehow conveyed to the other person. In addition, a mutual exchange of “how” tends to influence the substance of “what.” Consequently, whatever idea is described in the end is a result born out of a mutual exchange beyond words. For two people who have created such a relationship, both contribute to such an outcome.
Pursuing whether technology can really make people happy
—How far has your research progressed? How will it develop in the future?
In relation to haptic research, I have been working on haptic design principles and wrote the book “Tactility for Generating Information” (Fig. 4, recipient of the 69th Mainichi Publishing Culture Award in the Natural Science Category). I am also involved in a project to map onomatopoeia in industries specializing in the use of tactile sensations, based on the onomatopoeia distribution map in Fig. 3 (joint research with Professor Maki Sakamoto of the University of Electro-Communications). Preparing and sharing texture maps in this way can lead to mutual understanding and communication in relation to the sense of touch. Moreover, in 2016, I conducted interviews with lumber-related companies and trading companies specializing in paper and created onomatopoeia distribution maps compiling words that I obtained from those interviews. At present, these are maps corresponding to the products of companies involved in the interviews, but in the future, I think it would be beneficial to create such maps in collaboration with other interested companies and organizations.
I would also like to move forward on the research of haptic communication. We can somehow sense whether being touched by someone else is good and what kind of emotion the other person is experiencing. If this is the case, it should be possible to read human sensory information by combining that information with the sense of touch or smartphone vibrations. At present, there is a lot of research on individual sensations of touch, but in the future, I think the principles of combining tactile materials as in the case of music (combinations of sounds) will become clear.
In January 2017, I supervised the Japanese translation of “Positive Computing: Technology for Wellbeing and Human Potential” . This book describes how technology influences wellbeing and how wellbeing can be approached using technology. Today, society is overflowing with information technologies, and I would like to consider the results of accommodating and getting on well with those technologies instead of rejecting them outright and to then ask the question: “Can technology really make people happy?”
—Dr. Watanabe, can you leave us with a message for young up-and-coming researchers?
A sense of balance is vitally important for researchers. You should not research only what interests you. It is also important to choose something that others are not doing, that others are not good at, or that other people may enjoy. In the sense of creating something of value for society on the whole, it would be better to have someone else do that work if that person is more qualified than you. With this approach, you will be able to demonstrate your own abilities in an optimal way. I believe it is necessary to have both a passion for continuing what you think is important and a calm, objective mind when thinking about value and what it means to society and a particular research field.
Senior Scientist (Distinguished Researcher), Human Information Science Laboratory, NTT Communication Science Laboratories.
He received a Ph.D. in information science and technology from the University of Tokyo in 2005. His academic work has been published in scientific journals in the field of neuroscience and interface technologies. He has also presented his work at technology showcases, science museums, and art festivals such as at SIGGRAPH (2006°›2009, 2014) and Ars Electronica (2002, 2004, 2007°›2017). His research is focused on cognitive science and communication devices with applied perception.