Creating a Map or Encyclopedia That Anyone Can Use in Tactile Experience Design

Building a system of understanding that acts as a basis for designing tactile experiences

—Please tell us about your research.

The topic of my research is the “design science of tactile experiences related to the outside world and oneself.” “Tactile experience related to the outside world” refers to research that involves investigating the principles of human touch. That is, how human beings experience sensations like prickly or fluffy when touching an object. “Tactile experience related to oneself,” meanwhile, refers to research into investigating the principles of body awareness. For example, when someone touches an object other than their own body, what kind of object would feel like an extension of themselves.

I call tactile perceptions the “outside world” and physical sensations “oneself.” For example, when touching the tip of a fork with one’s finger, the skin on the end of one’s finger indents to the shape of the fork. This is an experience of one’s “own” body in which one’s skin indents and changes. At the same time, it is also a tactile experience with something from the “outside world” whereby the tip of the fork is pointed. I believe that the tactile experiences of oneself and the outside world are two sides of the same coin, and that it is important to research tactile sensations from both perspectives.

The term “design science” of human tactile perception and physical sensation means building a system of understanding for design that can be used when creating pleasant textures in product development and DIY (do it yourself), as well as when creating a body that feels natural as one’s own, such as a prosthetic limb or a metaverse avatar. When we hear the word “design,” we typically imagine the appearance and functions of a product or the design of a user interface, etc., but design in its essence is the act of conceiving something and bringing it to life. Design is a human behavior that everyone does regularly.

For example, when drawing a picture with a crayon, we think of what we want to draw and use a combination of different colored crayons to bring it to life. Colors primarily change depending on the wavelength of light, and physically, there are no clear boundaries, however, crayons are organized into a color system that is easy to understand such as red or blue. Crayons enable human beings to use a variety of colored crayons to achieve what they want to draw. I believe that building this crayon color system is design science.

Conceiving something and bringing it to life like this requires a system of understanding or a system of knowledge that compiles the elements or rules of this “something.” And this creating of a system of understanding that acts as a foundation or starting point to creating things is what I think of as design science.

—Please tell us about how you started your research and what interests you about it.

When I first joined the company, the first piece of research I engaged in was on the topic of touch. In this research, I took on the topic of speculating a person’s perceived texture or tactile sensation based on the movements of their hands and eyes while touching various objects. When turning my research into a paper, I created a map that systematically expresses the relationship between the movement of the hands or eyes and tactile perceptions such as roughness or hardness [1] (Fig. 1(a)). This map is a system of understanding for how to combine hand and eye movements to be able to estimate the desired tactile perception. It truly follows the philosophy of design science that I mentioned a moment ago. In hindsight, I feel this experience ties into what I am researching now.

From my desire to understand the experience of human touch in a holistic manner, I have gradually established my approach of creating a map. For example, I researched expressing the change in tactile perception in the velvet hand illusion (a tactile illusion that occurs when a person rubs their hands together with a mesh-like material such as a fine wire mesh or fabric held between them) on the tactile perception map [2, 3] (Fig. 1(b)).

After that, I learned the philosophy of design science from my group leader and realized that my interest was building a system of understanding that could also be used for design. I set about research that also considered contributing to design. I am currently researching a map [4] that expresses what combinations of textured materials create a pleasant tactile perception (Fig. 1(c)), as well as a map [5] that expresses tactile perceptions brought about according to simple pattern design rules (Fig. 1(d)).

Fig. 1. Examples of maps to systemize tactical relationships.

What is exciting about this research is that not only is this system of knowledge useful to design, but you can also enjoy the system itself. The words of Goro Yoshida, the first director of the Electrical Communication Laboratory (predecessor organization of NTT laboratories), are still referenced by many people today ever since the Ministry of Communications era. He said, “Do research by drawing from the fountain of knowledge and provide specific benefits to society through practical application.” My interpretation is that this “fountain of knowledge” is the insights of basic research that lead to practical implementation. In my research, the foundation of knowledge is the knowledge system that is acquired as a result of my work. I feel that this system of knowledge has the potential to provide benefits to the world, even if it is not yet at the stage where it can create some kind of product or service.

In fact, we often enjoy knowledge systems. The likes of encyclopedias, catalogs, and collection books are types of knowledge systems. They are well organized and interesting just to look at. In my work, the Illusion Forum qualifies as an encyclopedia knowledge system [6]. The Illusion Forum is a website that was published by NTT about 25 years ago where you can experience illusions (Fig. 2).

Fig. 2. Explanation of the tactile illusions published on NTT’s Illusion Forum website.

Up to now, it has provided over 130 pieces of content where users can experience visual and auditory illusions. I am also involved as a tactile perception researcher. In 2023, I collected various tactile illusions and published them as new content. While the Illusion Forum is used by many people for all kinds of use cases, including for educational purposes, we receive lots of feedback saying that it is also enjoyable just to gaze at. There are cases where people on social media try to explain mysterious experiences that have occurred in their day-to-day lives by referring to the Illusion Forum. This Illusion Forum acts as an encyclopedia or dictionary for illusions. The illusions discovered by our predecessors and the systemized versions of them spark intellectual intrigue, providing knowledge in a form that enhances the resolution of sensory experiences in daily life. The water drank from the “fountain of knowledge” is delicious as it is.

—Please tell us about your research environment.

My research does not require large equipment. However, because haptic display technology, equivalent to visual displays like monitors and auditory displays like speakers, has not yet been fully established, I use a more old-fashioned method in my experiments by repeatedly presenting various materials that people may encounter in daily life by hand in a laboratory setting (Fig. 3). In this experiment, I employed various restrictions such as moving the material at a constant speed, having the experiment participants be consistent with how they touch the materials, and limiting the touching time to five seconds. An issue with this experiment was the fact that I could not automate it. In an experiment the other day, I had a hard time as I had to repeatedly change the materials as many as 1600 times in one day.

Fig. 3. Engaged in velvet hand illusion research.

For the materials presented in the experiments, I mainly collected objects made of cloth, wood, metal, plastic, and rubber that we often touch in our day-to-day lives. There is not yet an established set of criteria or rules for selecting these materials. That is why, while referencing the materials that were investigated in the previous research, I search and collect materials myself in 100-yen shops and prepare them in shapes and thicknesses that can be used for experimentation.

I want to be able to design worlds in which we can feel our body and objects through our body

—Please tell us about what you have achieved so far, your future challenges, and what you wish to do in the future.

As well as the tactile perception map, I am also creating a physical sensation map. This is joint research with Professor Kenri Kodaka of Nagoya City University. Together we investigate the tactile perception of materials that cause the physical illusion of a paralysis illusion [7]. A paralysis illusion is a physical illusion whereby if you press your own fingers against someone else’s and then pinch and rub them with your other hand, the other person’s finger starts to feel like your own and even seem numb, as if they were paralyzed (Fig. 4(a)).

It has been discussed that the tactile perception of another person’s fingers might be important for the physical illusion, as this illusion is less likely to occur when a pen is attached and rubbed instead of another person’s finger. I thought that if I could discover specifically what kind of tactile objects tend to cause this illusion, I could identify the tactile perception that can cause an illusion of being one’s own body. That is why we switch the other person’s fingers with objects of various tactile perceptions and investigate the strength of the illusion (Fig. 4(b)). The map produced as a result expresses what kind of materials invoke a strong illusion. From the map, you can see that the red zone of soft and smooth materials is more likely to cause a strong illusion (Fig. 5).

Fig. 4. Investigating a physical illusion setup and the feeling of materials that can provoke physical illusions.

Fig. 5. Map that expresses the feeling of materials that tend to trigger physical illusions.

I also think this research is useful as a map for designing human-like tactile perceptions. For example, I believe the insights of this research can be used as a material map when wanting human skin-like texture for attachments to one’s body, such as prosthetic arms, or for humanoid robots. Conversely, it can also be used in cases where someone wants products that are free of human-like qualities.

Going forward, I wish to pursue the question of whether we can design desired tactile perceptions and physical sensations and a “world you can feel” by using and combining new technologies in various ways.

For example, if we achieve Digital Twin Computing, aimed for by the Innovative Optical and Wireless Network (IOWN), and become able to simulate the world as perceived by others or non-human creatures, we might be able to relive their experiences. What’s more, greater high-capacity and high-speed communication could result in us experiencing events happening on the other side of the Earth or even on the Moon in real time. In other words, new technology can be expected to generate new physical sensations and new world experiences. What is key here is people being able to choose the tactile perceptions they want to experience, the body they aspire to have, and the world they wish to immerse themselves in. Therefore, I want to create a future where anyone can easily design based on a recipe book in which tactile perceptions, physical sensations, and the “world you can feel” are described systematically.

To expand on the topic of “a world you can feel,” the world we see and feel changes according to the shape and functions of our bodies. For example, supposing new technology allows us to have wings. As a result, advertising signboards in high places or roofs that you hadn’t paid attention to until now will appear at your feet or in places that should be avoided. You will probably no longer be conscious of steep hills or stairs that you were aware of until now.

By your body changing in this way, the way you interact with the world will change, and ultimately, how you experience the world will change significantly. Video games continue to create new worlds that tie in such physical functionality. In a certain video game, the player has the ability to break through the ceiling at their own will. Interestingly, people who have experienced this video game sometimes, in the real world, when casually stepping out onto the balcony and looking at the ceiling of the floor above, suddenly think to themselves, ‘Oh, I could go through here.’ It gives new meaning to a ceiling they had never paid attention to before. There are lots of video games in the world, including this one, that create virtual worlds that offer extremely diverse and free interpretations.

I believe that if I can understand the elements that make up these worlds and systemize them, it will be useful in designing the worlds you want to immerse yourself in in the future. I have also started to undertake research into the experience of video game players, involving members of my research group and other personnel from the Osaka University of Arts. In particular, we have started to produce specific results in the analysis of remade video games, conceived by the group’s Lead Researcher Kazushi Maruya [8, 9]. Remade video games are a category of video games where the world of the original work is recreated using modern technology.

As a result of investigating remakes of existing video games, we learned that being able to deeply project oneself onto the protagonist and simulate the consequences of one’s actions in the real world are important factors in worlds produced by video games. Going forward, I want to make a recipe book for creating the worlds we desire, by systemizing these factors and turning them into a map.

As far as research challenges, as the goal of my research is to “create a system of knowledge that can be understood by everyone,” I feel that how to create a map that can be understood instinctively is a major challenge. While Fig. 1 shows tactile experience in a two-dimensional map, in actual fact, the experience of human touch is more multi-dimensional. However, because it is difficult for human beings to directly understand multi-dimensional spaces, I think that we must identify the essence that we can comprehend and simplify it.

—Do you have a message for young researchers, students, and business partners?

There are limits to what I can do alone or what can be done in my field alone, so, if possible, I want to work with people who are involved in things I do not know. Focusing only on one’s area of expertise or similar fields will see your way of thinking become rigid, and there is a possibility that problems that cannot be resolved with your skillset are left unresolved. I think that the concept of design science, which I mentioned earlier, is to borrow everyone’s strengths to extend the subject into various fields and connect it to unexpected applications so to avoid becoming self-centered.

I think it is beneficial for students to experience not only their area of expertise but all kinds of fields. I am also involved in the Folklore Society of Japan, attend various workshops, and read papers and books of all kinds of fields. I have experienced firsthand that the perspective expands by stepping into different fields. In a way, it can be said that the world we see changes by getting a new body in the form of knowledge. If your world becomes bigger, you will be able to do unique research that only you can do, and this will lead to you acquiring strengths unique to you that no other researcher has.

References

[1]	T. Yokosaka, S. Kuroki, J. Watanabe, and S. Nishida, “Linkage between Free Exploratory Movements and Subjective Tactile Ratings,” IEEE Trans. Haptics, Vol. 10, No. 2, pp. 217–225, 2017. https://doi.org/10.1109/TOH.2016.2613055
[2]	T. Yokosaka, S. Kuroki, and S. Nishida, “Describing the Sensation of the ‘Velvet Hand Illusion’ in Terms of Common Materials,” IEEE Trans. Haptics, Vol. 14, No. 3, pp. 680–685, 2021. https://doi.org/10.1109/TOH.2020.3046376
[3]	T. Yokosaka, S. Kuroki, and S. Nishida, “Technique for Modulating the Tactile Sensation of Objects Using an Illusion,” NTT Technical Review, Vol. 19, No. 9, pp. 46–51, Sept. 2021. https://doi.org/10.53829/ntr202109fa5
[4]	T. Yokosaka, “Textural ‘Marriage’: Tactile Pairing Principles for Everyday Materials,” IEEE Haptics Symposium 2024, Long Beach, USA, Apr. 2024.
[5]	T. Yokosaka, R. Sumida, T. Ohtani, and K. Maruya, “Aesthetic and Affective Dimensions Explaining Variation in Tactile Perception of Periodic Patterns,” Eurohaptics 2024, Lille, France, June/July 2024.
[6]	Illusion Forum website (in Japanese), https://illusion-forum.ilab.ntt.co.jp/
[7]	T. Yokosaka, Y. Sato, and K. Kodaka, “Tactile Properties of Everyday Materials for Inducing a Body Illusion,” Eurohaptics 2024, Lille, France, June/July 2024.
[8]	T. Yokosaka, K. Miura, Y. Isogaya, T. Ohtani, and K. Maruya, “Factors of Player Experience in Describing the Relationship between Remade and Original Works,” 2024 IEEE Conference on Games (CoG), pp. 1–4, Milan, Italy, Aug. 2024. https://doi.org/10.1109/CoG60054.2024.10645618
[9]	K. Maruya, T. Yokosaka, Y. Isogaya, and T. Ohtani, “Factors in the Playing Experience That Are Important for the Evaluation of Remade Video Games,” Proc. of the Entertainment Computing Symposium 2024, pp. 255–260, 2024 (in Japanese).