Scientific Understanding of the Desire to Touch and Haptic Presentation Methods Based on Cross-modal Perception
—Toward New Horizons in Haptic Content That People Really Want to Touch

Clarifying the “desire to touch” from large-scale social media data

—Dr. Ujitoko, what kind of research is involved in obtaining a “scientific understanding of the desire to touch”?

“Scientific understanding of the desire to touch” refers to research that takes a psychological approach to understanding such riddles as “What kind of things do people think they want to touch?” and “Why do they think they want to touch them?” By promoting this kind of research to deepen our understanding of the “desire to touch,” which has not been studied much in prior research, could we not provide the knowledge needed to create haptic content that people want to experience?

In contrast to the sense of touch, information presentation technologies involving the sense of sight and sense of hearing have become widely used. Today, the world is overflowing with attractive and compelling music and video content, and each of us can experience content that we want to experience. However, while the research and development of haptic presentation technologies involving the sense of touch have made some progress, the current situation is that haptic content that a person wants to experience has yet to spread in society. I believe that a major reason for this is a lack of understanding as to what kind of haptic experience people want. In the past, investigations into the desire to touch consisted of experiments conducted in laboratories with subjects using actual stimuli. These laboratory experiments, however, could only use a limited number of targets from among those we touch in our daily lives, and could also produce biased results since subjects were conscious of results that the experimenter was seeking. For these reasons, it was difficult to investigate the desire to touch in everyday life.

We collected a large amount of text related to the desire to touch in everyday life stored in social media and aimed to understand the desire to touch in a comprehensive manner from the viewpoint of diverse situations in daily life. By clarifying what people want to touch in what kind of situation as basic research based on large-scale social media data, it would be possible to obtain insights into haptic content that people want to experience and to identify where we should focus our efforts in developing technology based on those insights. For example, if it’s understood that the desire to touch cats in one’s living room on one’s day off is a strong desire, a decision can be made to prioritize the development of technology that presents a feeling of cat fur and paw pads, which may then lead to the creation of content that people want to experience in the future (Fig. 1).

Fig. 1. Process of creating haptic content based on knowledge of the desire to touch.

—What have been some concrete findings in your research to date on the desire to touch?

In collecting data related to the desire to touch, we found that targets that people would like to touch are extremely limited in number. This was an unexpected result since there are a limitless number of targets that we can touch all around us, which suggests that there would be a wide variety of targets that people would like to touch. In demonstrations presenting the results of certain studies on haptic presentation, the ability to express the feel of all sorts of targets is often regarded to be a strong point and an attractive feature of those studies. Our above result, however, suggests that the ability to reproduce certain types of tactile sensations may be appealing even if the feel of a wide variety of targets cannot be expressed. We also came to understand that the targets that people want to touch are biased on the whole towards living things like people and animals over things. For this reason, I believe that content that can provide communication through remote touching of a loved one or family member or through touching of virtual avatars in the metaverse, or content that can produce a calming effect by touching animals like cats and dogs, may in the future become an integral part of human life. In addition, we found that there is a strong relationship between the target of what a person wants to touch and the way of touching that target. For example, “stroking” and “cat” are strongly related, so when creating “cat touching” content, it may be best to start out by formulating a policy that places importance on expressing the feel of stroking.

This research began during the spread of COVID-19 when I thought to myself “Could this be research that I could conduct from home?” As it turns out, I also found that the desire to touch was changing during the spread of COVID-19. From around the time of Japan’s Declaration of a State of Emergency in response to the COVID-19 outbreak in 2020, I obtained a result suggesting that the phenomenon of “skin hunger,” or a craving for the warmth of the skin of living things like people and animals, was appearing among people. I thought that this was likely due to the effects of social distancing and restrictions on going out on the desire to touch (Fig. 2). This result led me to conclude that there was a need for technology that could achieve a tactile connection with other people in a remote environment during the COVID-19 pandemic. Similarly, since this research also identified the desire to refrain from touching non-living things like cash or door knobs thought to be a source of infection, I found that technology that could provide a sense of touch even when not actually touching something would be needed during such a pandemic. In addition, a series of our studies revealed that the desire to touch possessed not only static properties but also dynamic properties that change depending on the event. This was a deeply interesting discovery that reflects the complexity of the desire to touch.

Fig. 2. Change in desire to touch and COVID-19 outbreak.

—How are you researching “haptic presentation methods based on cross-modal perception”?

I am now studying haptic presentation methods based on cross-modal perception as technologies that can give people a sense of touch. To give an example of cross-modal perception, when a person is eating red-colored kakigori (shaved ice, a summer treat in Japan) that smells like strawberries, a phenomenon occurs in which that person experiences the taste of strawberries despite the fact that kakigori is essentially tasteless. What happens here is that perception corresponding to a certain sensation is modulated by input into a different sensory organ. Now, if we were to apply this phenomenon to haptic presentation when touching something with one’s finger, we could produce a pseudo sense of touch by simply changing the way in which the finger and target movement is viewed as visual information without any physical haptic stimulation. In research using this technology, I am proposing a haptic presentation method associated with the swipe operation on a display in which the user perceives “heaviness,” which should be absent in such a display operation, by limiting the amount of scrolling (Fig. 3).

Fig. 3. Example of a haptic presentation method based on cross-modal perception.

Past haptic presentation devices were often expensive and difficult to carry around, so in actuality, spreading their use in the manner of “one per household” was deemed difficult. In the face of this problem, I considered that applying cross-modal perception to haptic presentation could lead to a device based on a simple and inexpensive mechanism.

In my studies of haptic presentation methods based on cross-modal perception, I have not only proposed individual presentation methods but also undertaken the systematization of presentation methods. In this regard, a variety of methods have previously been proposed as haptic presentation methods based on cross-modal perception. For example, various methods have been proposed such as a method that expresses a feeling of heaviness when picking something up and a method that expresses a feeling of roughness when rubbing the surface of an object. In the past, however, knowledge associated with these methods (such as the method itself and its limitations) was presented in separate papers in a fragmentary manner. In such an environment, researchers and developers of haptic content cannot efficiently obtain knowledge. For this reason, I conducted a comprehensive survey on related research from the 2000s up to now and organized haptic presentation methods from viewpoints of tactile feeling, user input, and visual stimulus. This type of work is similar to that of a consultant in terms of the need for conducting a comprehensive survey and organizing information based on meaningful viewpoints. However, this requires working in a way different from that of a researcher, so completing this survey was a challenge for me. This survey clarified the present state of “haptic presentation methods based on cross-modal perception,” which have contributed greatly to the field of haptic research. The survey also enabled me to discover topics missing in the field to date.

Creating appealing haptic content and generating a paradigm shift in the haptic field

—What are some difficult points in your current research?

In analyzing the desire to touch using large-scale social media data, it is difficult to determine whether information on the desire to touch can truly be obtained as a result of investigation. As a research theme, the desire to touch crosses into a person’s private domain, but on the other hand, social media data constitutes information that the user wants to publically release. So we need to be careful in deciding whether publically released data truly reflects the desire to touch. In addition, there is a bias with respect to user gender or age depending on the social media, so a bias naturally arises in results. In my research, I deal with these problems by comparing the results of collecting large-scale data in online experiments with the results of social media data to clarify as much as possible the bias originating in the method of data collection. In my research to date, I have come to understand the targets of the desire to touch, but I have not yet been able to clearly identify what kind of tactile sensations should be presented in those targets or what kinds of situations are more desirable. My research objectives from here on are to clarify “What kind of tactile sensation is needed in a target that people desire to touch?” and “In what kind of situation is a touching experience needed?”

Another problem in a haptic presentation method based on cross-modal perception is that the degree of feeling differs from one person to the next. In actual demonstrations targeting a variety of people, we have found that the way of feeling changes depending on the person and that the ease of feeling differs according to age. By clarifying the factors that give rise to such differences in the way of feeling, I hope to apply the knowledge gained to calibration techniques in haptic presentations to mitigate individual differences. Moreover, in comparison to past haptic presentation devices that physically stimulate a person’s body, a problem in existing haptic presentation methods based on cross-modal perception is that the tactile sensation that can be felt is very weak. To solve this problem, I will combine existing methods with physical stimuli based on simple haptic presentation devices, and from here on, I plan to search for haptic presentation methods based on more powerful means of cross-modal perception.

—What do you think will become possible in the years to come based on your research achievements?

I believe that using the results obtained from the scientific analysis of the desire to touch and from haptic presentation methods based on cross-modal perception should be able to contribute to various elements making up the Innovative Optical and Wireless Network (IOWN) vision proposed by NTT. For example, in Digital Twin Computing, one of the three key elements of IOWN, our results can contribute to making the digital twin itself more appealing by implementing in the cyber world “content that people desire” born from an understanding of the desire to touch. In addition, haptic presentation methods based on cross-modal perception can be used as a mechanism for conveying to a user in the physical world a haptic experience of the user’s digital twin in the cyber world. In this way, it will be possible to provide a natural haptic experience without special devices.

—Dr. Ujitoko, what message would you like to leave for researchers, students, and business partners?

I myself have been in this research field related to the sense of touch for some time, but the present situation is that the general public knows little about this research with no clear outlook for the spread of haptic content in society. This can make me feel a bit sad as a researcher. To overcome this state of affairs, what motivates me in my research is a desire to produce results that can support technologies and content that the general public can easily use while pursuing my work in basic research. From here on, as well, I will focus on basic research that could lead to appealing technologies in the future. Furthermore, in contrast to a desire for immediate profits, I hope that the knowledge that I accumulate in relation to the sense of touch will enable someone in the future to gain some valuable insights. To those readers who have an interest in the research that I am pursuing here at NTT Communication Science Laboratories, I would like to say, “Let’s work together to create a new future for haptic research!”