Information-presentation Technology That Gives an Illusory Texture of Objects without Needing to Touch Them

Using illusions to manipulate the texture of objects without needing to touch them

—What research are you currently conducting?

I’m investigating information-presentation technology that uses illusions. I believe that by using illusions, we can perceptually present information that is physically difficult to present. In the previous interview (November 2020 issue) [1], I introduced our information-presentation technologies that use projection mapping, namely, “Hengento,” which gives an illusion as if a stationary object is moving and “Ukuzo,” which gives an illusion as if a stationary object is floating in the air. I’m currently working on two themes: “technology for manipulating the texture impression of virtual objects for touchless user interfaces” and “tangible interfaces and virtual interfaces.”

Regarding the first theme, I’m investigating methods for more directly manipulating the texture perception of objects that I briefly touched upon in the previous interview. Together with my research colleagues Takumi Yokosaka and Yusuke Ujitoko, I studied the problem of how to convey the weight of an object on a screen, which cannot be touched directly, to a user when they make a lifting motion. In an experiment, we moved the stripes displayed on a screen in synchronization with the user’s lifting motion. When the speed at which the stripes moved was decreased, the user felt that the stripes were heavier, even though no force was actually applied to the user’s hand. We also found that the direction of movement of the stripes does not necessarily have to coincide with the direction of the lifting motion; for example, the user feels that the stripes are heavier even if the speed of the stripes moving horizontally is decreased to match the upward lifting motion (Fig. 1). I expect that by applying these research results, we will be able to give a sense of weight to virtual objects that a user manipulates without needing to touch them in a manner that creates a highly realistic impression of the manipulation.

Fig. 1. Illusion of weight.

As shown in Fig. 2, when the user holds their hand in front of the laptop and moves it left or right, the textured virtual object in the upper left corner of the screen appears to stretch. The amount by which the virtual object is stretched changes in accordance with the amount by which the user’s hand moves, and more stretching makes the object feels soft and less stretching makes it feel hard. If the hand goes out of the range detected by the camera and the hand movement is not reflected in the virtual object midway through the movement (the state of being stretched stops midway through the movement), something felt strange to the user, but we did not understand the reason for that strange feeling in details. Therefore, Yusuke Ujitoko and I set an effective range (i.e., detection range of the camera) in which the sensing function works for hand movements that stretch the virtual object and investigated how the impression of the virtual object changes in accordance with the effective range. We found that when the stretching of the object is stopped midway, the object feels hard, heavy, and has friction. This result indicates that engineers who create touchless devices must understand the effect that the effective range of hand movement for manipulating a virtual object has on the impression of the texture of the object. Conversely, it also indicates that by using the effect of the effective range, they may be able to provide people with additional texture impressions of an object.

Fig. 2. Changes in the impression of stretching.

We submitted many papers to academic journals on technology for manipulating the texture impression of virtual objects for touchless user interfaces, and our latest research has been accepted for publication in a high-impact-factor journal, the IEEE (Institute of Electrical and Electronics Engineers) Transactions on Visualization and Computer Graphics [2]. Under the theme of moving robots in a touchless manner, I also started researching causal perception and texture perception when moving objects by moving a cube-shaped robot through hand movements.

Regarding the second theme “tangible interfaces and virtual interfaces,” using scientific understanding of human characteristics and physical properties, I aim to develop a next-generation user interface that supports interaction between people and information. I started this theme in October 2022 as research on a new form of user interface that fuses tangible interfaces and virtual interfaces. Tangible interfaces enable users to access information while touching real objects and intuitively interact with information as an extension of their daily lives. They, however, use real objects, so they are subject to time and physical constraints. On the contrary, virtual interfaces enable users to access information virtually, so time and physical constraints are fewer; however, it is more difficult for users to interact with information as an extension of their daily lives because they have to wear a head-mounted display or use unfamiliar devices. Accordingly, I imagine a new user interface that merges tangible and virtual interfaces by, for example, adding a tangible interface to the cross reality (XR) approach. I am considering tangible interfaces as one solution to the question of how to provide natural XR experiences for people.

—So, letting users feel the texture of objects in a touchless manner makes it possible to provide them with a more convenient and enriching experience, right?

We have shown the possibility of presenting textures, such as weight and hardness, of objects by combining hand movements and video. We are now entering a research phase of increasing the reality of the presentation. I believe that in the future, our research results could be applied to present information in online-shopping and other applications. For example, you may have heard stories of how the firmness of a pillow or the feel of clothes bought online differed from what you imagined. Although the use of video can express a feel of a material to some extent, it has yet to accurately convey the texture to the potential buyer. Further development of information-presentation technology that conveys the texture of objects will make it possible to correctly convey information, including texture, to buyers so that they can buy items that match the ones they imagine. Conveying textures, such as the warmth of skin and feel of palms with family members living in remote locations makes deeper communication possible. By linking texture-manipulation technology using touchless interfaces with appropriate visual-expression technology, information presentation using illusions can be made richer and more elaborate.

Deepening research through the integration of psychology and engineering

—You originally majored in psychology, right?

In Japan, illusions and perception are studied in the field of psychology. We are currently using a psychological approach to explain the phenomena concerning illusions and perception and applying the phenomena to everyday life through integration with engineering technology. Since psychology is the study of people, psychological experiments take time, and unlike mechanical systems, people do not always respond in exactly the same way to the same stimuli, interactions, etc. It is therefore difficult to reproduce and collect data, and it also takes time to build theories using the collected data. Research on engineering is evolving at a dizzying pace, so how to close the gap in speeds of these two research fields will be the key factor in research on illusions and perception. Artificial intelligence (AI) has developed remarkably, and I believe that by using AI, we can close this gap.

If it were possible to create a complete human model and analyze it using Digital Twin Computing, which uses AI, I believe the above problem could be solved. However, people interact with each other and react differently according to the environment; therefore, it would not be straightforward to create a human model by taking all the data and training the model using them.

In psychology, we analyze elements of human perception separately, for example, visual perception, auditory perception, and interpersonal perception. I believe it is possible to use AI to model each element, and, in fact, such efforts have been made. By fusing these models, we will be able to explain the full picture of the human mind.

Scientifically understanding the psychological characteristics of the recipients of information is also important in the field of user interfaces. I believe that scientific research concerning user interfaces can only be completed by fully understanding how the nature of an object to be conveyed can be expressed via an interface and why it is conveyed to users.

Break the mold and create new values by integrating the values of others with your own values

—What do you keep in mind as a researcher and what do you aim to achieve in the future?

I think that researchers tend to get caught up in their own logic and think that their values are correct when they immerse themselves in their research. If you continue to conduct your research while being confined to your own values, you may begin to struggle as you are not really sure if you are doing the right thing or hit a wall. I became a group leader three years ago and have had many opportunities to learn about the values of others in various ways. Of course, sometimes I feel that the values others hold do not match those of mine; regardless, I have recently come to strongly believe that it is important to understand and recognize the values of others then create new values by integrating their values with my own and that by doing so, I can grow as a person.

For example, the speed of recent academic research on AI is remarkable, and it is not uncommon for a problem that could not be solved yesterday to suddenly be solved today. The researcher working on that problem must look for their next research target. In that case, some researchers may lose sight of their goals because their research field suddenly becomes obsolete or others may shortsightedly set easy targets in an attempt to stay just slightly ahead of new technologies. Therefore, it is crucial to acquire knowledge in multiple fields, connect a variety of knowledge, and expand your own research area. To that end, it is important to create new values by merging the values of others with your own. Even if there is one answer to a problem, there is often more than one way to solve that problem. The speed and effectiveness of each solution will differ, and in some cases, combining several solutions may create synergy. By combining several solutions, you may also discover new problems. I believe that learning and combining knowledge from multiple disciplines to solve problems will naturally expand your research area and your capability as a researcher will broaden.

I hope to continue to be a researcher, but I want to be a researcher who can accommodate and combine diverse values and continue to create and provide new values. As I create new values, I may look at other areas or themes. I think that even in such a case, I always have something to rely on as a starting point, and by using that point as a foundation, I can make great progress. My foundation and starting point is psychology, and I intend to absorb various new knowledge on the basis of psychology. I want to continue my research so that I can successfully integrate psychology and engineering and contribute to society.

—Do you have a message for younger researchers?

I know that your superiors and peers expect a lot from you. It is, of course, important to meet those expectations, but also exceed those expectations and break the mold. Since meeting expectations means staying within the range imposed by your superiors and peers, you will inevitably have less scope and fewer opportunities for growth. I believe that the moment you exceed those expectations, you will have more discussions with your superiors and peers. By having discussions, you can involve the people around you and move the whole process forward. To do so, it is necessary to have a foundation, or something to rely on, which may be knowledge including your own experience, knowledge of others, or accumulation of such knowledge. Let us exceed the expectations of those around us, go beyond our imagination, and move forward together.

References

[1]	T. Kawabe, “Do Your Best to Be Second to None in Achieving What You Want to Do,” NTT Technical Review, Vol. 18, No. 11, pp. 6–10, Nov. 2020. https://doi.org/10.53829/ntr202011fr1
[2]	T. Kawabe and Y. Ujitoko, “Softness Perception of Visual Objects Controlled by Touchless Inputs: The Role of Effective Distance of Hand Movements,” IEEE Trans. Vis. Comput. Graph., Mar. 2023. https://doi.org/10.1109/TVCG.2023.3254522