Do Your Best to Be Second to None in Achieving What You Want to Do

Applying illusions in the real world

—Please tell us about the research you are currently conducting.

My current research theme is information presentation using illusions. Illusion refers to the perceptual characteristic of humans in perceiving something different from what is happening in the real world. Such an illusion is generally thought of as undesirable because we think it is preventing us from accurately seeing the real world. However, thinking that using an illusion makes it possible to create a new way of expression and provide a sensory experience that is unlikely to occur in the real world, I and other researchers at NTT Communication Science Laboratories developed a technique called Hengento (deformation lamp) [1] (Fig. 1). This technique was also the trigger for initiating my current research theme.

Fig. 1. Hengento.

Using projection-mapping technology, Hengento makes it possible to give an illusion in which a stationary paper object appears to be moving. Hengento projects an image of light and dark calculated to create an illusion that exactly overlaps the target. The mechanism for detecting movement in the human brain works even in the case of low contrast, so the image projected by Hengento may also have low contrast.

Human motion detectors are sensitive to brightness but insensitive to color; therefore, a change in brightness is sufficient for the image projected by Hengento. Since only a faint light-and-dark image is projected, it is possible to give the illusion of movement only to the target without losing almost any of the color or texture of the projected target. This technique was commercialized in collaboration with NTT Communications and Dai Nippon Printing Co., Ltd. and is being used in various places such as supermarkets and museums.

I then developed a technique called Ukuzo (Fig. 2) [2]. By projecting a shadow pattern onto a real object, Ukuzo gives an illusion as if the real object is floating in the air. It has long been known that an object appears to float up from its background when a shadow is cast onto it, and this phenomenon is used in applications ranging from computer interfaces, comic books, and animation. By simply capturing an image of the target object with a camera, Ukuzo automatically generates and projects a shadow pattern of the object onto the object and provides an illusory depth impression of a real object.

Fig. 2. Ukuzo.

I also worked on Danswing Papers (danswing is a combined word of dance and swing) [3] (Fig. 3). It has been known for a long time that if light and dark lines are added to the outline of a stationary object, and the object is presented on a background with temporarily changing brightness, the illusion of the object moving is created. However, it has only been applied to displays. I have reported that adding a light and dark outline to a picture on a paper object and placing the object on a display that switches between light and dark gives the impression of the picture moving. That report was highly evaluated in the Best Illusion of the Year Contest at an international illusion contest in 2018 [4]. I am currently working on the commercialization of this technique.

Fig. 3. Danswing Papers.

—All three are unique and exciting techniques. What are you currently focusing on?

For about a year and a half, I have been researching illusion techniques that convey visual “softness.” Softness is mainly perceived through the sense of touch, and how humans perceive softness only from images is not completely clear.

With this issue in mind, I focused on a physical phenomenon called the Poisson effect. The Poisson effect is a physical phenomenon by which a material contracts vertically when pulled horizontally (or expands vertically when squeezed horizontally). It can be described by a numerical value called Poisson’s ratio. For example, Poisson’s ratio of rubber is close to 0.5, but that of cork is almost zero. In other words, if rubber is stretched horizontally, it will shrink vertically, but if cork is stretched horizontally, it will hardly shrink vertically. Poisson’s ratio is thus a good indicator of such a physical property. It has been shown that Poisson’s ratio for everyday materials (excluding metamaterials) does not exceed 0.5.

I am researching how Poisson’s ratio contributes to the appearance of softness of objects. First, I investigated how humans perceive Poisson’s ratio and found that humans do not feel uncomfortable with an object when its Poisson ratio exceeds 0.5. As a result of repeated investigations, I also found that humans judge discomfort due to the Poisson effect based on the change in the area of an image before and after the image is deformed.

More recently, I showed that the Poisson effect affects the human pseudo-tactile sensation. Pseudo-tactile sensation refers to the haptic illusion generated in a user’s mind when a visual image is manipulated. For example, if the cursor suddenly slows while you are moving your computer mouse, you get the impression that it is heavy. The impression is considered a kind of pseudo-tactile sensation. I created an experimental setting that enables an object to extend laterally on a display while a user tries to horizontally extend an imaginary object in the air. I then tried to manipulate Poisson’s ratio. As a result, I found that the higher the Poisson’s ratio, the softer the object is perceived. I think in the future this technique can be applied in scenes in which the softness of objects in remote areas can be conveyed and controlled.

The trigger was a movie that I watched on a flight back from an academic conference

—Could you tell us what inspired you to carry out these studies?

When I attended an academic conference held overseas in 2018, the idea leading to this technique came to me in a flash when I watched a movie on my return flight. I think the movie was “Iron Man,” and in the movie, the main character was operating a three-dimensional hologram. In the real world, holograms are just images, so you can’t touch them; even so, I imagined that the hero had the sensation of touching the hologram and was operating it. Therefore, I started my research thinking that the texture of an object could be enhanced by giving it a pseudo-tactile sensation.

When I think back to my childhood, I felt like I could see through my hand when I held my hand over a miniature bulb, and I wondered if it was an extrasensory perception. As I grew older, I gradually became aware of reality, and realized things that do not actually exist may look like as if they existed. That realization might be the original experience that leads to current research activities. Some time after that, I majored in psychology at university and studied vision science. Moreover, as a fan of video games, I felt the impact when I crashed into a wall in the car racing game, so I wondered if it would be possible to transmit information using such a visual illusion.

—Have you personally changed since becoming a senior distinguished researcher? Please tell us what you have in mind when looking for a research theme.

It’s only been two months since I became a senior distinguished researcher, and I’ve been working from home to help prevent the spread of the novel coronavirus, so I haven’t yet felt a change in my role. I did begin to think about the viewpoint I should have. When I was working at a university 10 years ago, I became determined to contribute to science by becoming a science building block. However, I started thinking that I could pursue research differently at NTT than at a university. After starting to conduct research at NTT on the application of visual illusions in the real world and having people use those illusions, I felt that my research can not only contribute to the foundation of science but also to society. As a senior distinguished researcher at NTT, I’d like to be a person who can provide valuable research results to both science and society.

I think that finding research themes is one of the aspects that researchers will struggle with. Even now, I sometimes wonder, “What should I do next?” I remember when I was a graduate student, I was worried about finding a research theme, and I was dressed down by a senior student, who is now a university teacher, saying, “You cannot decide what to do because you are not making enough effort.” Certainly, if you don’t structurally understand the issues in your research area, you won’t know what to research. Thus, the resulting words “You’re not trying hard enough.” uttered by my senior were apt. To understand the issues in my research area and what I do not know, I try to read four or five papers on themes in that area during my spare time every day.

I also try to imagine things in the manner of the song of the Japanese cartoon character “Doraemon,” namely, “That looks fun; I wish I could do it.” For example, first, I imagine that it would be great if we could use illusions to do such wonderful things. I then think whether it would be feasible and what state-of-the-art knowledge and technology are needed to achieve this. When I ponder all this, I can lay out specific research processes, and the papers that I’ve been reading daily will be useful. In this way, I think that proceeding with research from the bottom up (namely, understanding what is unknown in a particular research area) while continuing to think from the top down (namely, thinking what we can achieve in the world) will make it easier to find issues and research themes.

Certain events that happened at graduate school influenced how I settled on research themes and what approaches I took. When I was worried about my career path, a professor in the laboratory next to mine said, “Mr. Kawabe, you can do as much research as you like, not as much as you need to do, but as much as you want to do.” After hearing those words, I realized that someone else will probably do what I need to do, so I’ve been continuing researching what I want to research without losing out to anyone. Those words are the reason my research got off to a good start.

I’m constantly thinking that I shouldn’t limit myself to a particular research field. To solve problems in the real world, knowledge of one research field is rarely enough; instead, in most cases, knowledge of multiple research fields is required. In such a case, I will not easily give up solving the problem even if it’s not my field; instead, I’ll try to acquire knowledge by reading papers in other fields. However, it is also true that it is impossible to surpass experts, so I try to listen actively to people in other fields. I think that the accumulation of knowledge will facilitate discussions and future collaborations.

Take up challenges with bottom-up and top-down approaches

—Please say a few words to junior researchers.

In my case, since I still have more seniors than juniors, I’d like to say what is important to us as I stand in the same shoes as my juniors. I’d like you, our juniors, to keep in mind bottom-up and top-down approaches that I talked about earlier. Taking these two approaches, you will naturally see how you can contribute to your research area as well as to society. I think it’s fine to focus on research themes that no one else has tackled if you want to. There is always someone to appreciate your desire if you try. My hope is that many creative and outstanding researchers will be nurtured in doing so.

What’s more, I think it’s probably not so easy for researchers to link their basic research to direct output. For example, we put Hengento into practical use after receiving feedback from someone who looked at our research results and suggested we could use it in a certain way. I also think that researchers who take a top-down approach and have aspirations can lead to output. In this sense, NTT provides good opportunities such as NTT R&D Forum, so I think it is important for our juniors to actively take those opportunities to obtain the opinions of others. With that in mind, I am actively exhibiting my research results at every opportunity.

—How will you move forward? Please tell us about your challenges and aspirations.

My future challenge is to expand the real-world implementation of illusions. Illusions occur not only with regard to vision but also to other senses such as hearing and touch. There are also illusions that occur when multiple senses combine. I think that we can change the scope and depth of technology that uses sensations by incorporating illusions that occur with various sensations into that technology. For example, I believe that we can conduct research from various perspectives under the keyword “softness,” namely, expressing softness through a combination of visual and tactile senses, manipulating the softness of real objects by using light projection, and judging softness in virtual space.

Now that I’m at NTT laboratories, and I’m often asked why I’m doing certain research. I feel that my research stance differs from that when I was researching at a university—where I had little contact with the outside world. I have come to think that my research is often questioned because researching at NTT laboratories may contribute to society. I feel that it is important to make not only scientific contributions but also social contributions by using the technologies that I have researched. After the state of emergency was declared in Japan to prevent the spread of the novel coronavirus, I wondered what kind of social contribution I could make to help stop the spread. For example, aiming to promote non-contact technology, thereby prevent infection, we are trying to develop a display that uses illusion to give the user the sensation of touching the buttons of a cash machine, elevator, or the like.

I think it is vital to solve these social problems one by one and anticipate problems that may occur in society. I had been thinking about the idea of a non-contact display even before the coronavirus crisis and was attempting to propose a rich and profound information-presentation technique by using illusions. Such non-contact displays could help prevent the spread of infectious diseases.

I imagine that people might not understand exactly what researchers like me are doing; however, I also believe that people think researchers are trying to improve society by gathering wisdom. I don’t know anything other than my work as a researcher, and I’ve never wanted to be anything else. I’ll continue researching what I like by keeping the two approaches of top down and bottom up in mind.

References

[1]	Video of Hengento, https://www.youtube.com/watch?v=OmdSxdSs-w4
[2]	T. Kawabe, “Ukuzo—A Projection Mapping Technique to Give Illusory Depth Impressions to Two-dimensional Real Objects,” NTT Technical Review, Vol. 16, No. 11, pp. 30–34, 2018. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201811fa4.html
[3]	T. Kawabe, “Danswing Papers,” exhibition panel of NTT Communication Science Laboratories Open House 2019, http://www.kecl.ntt.co.jp/openhouse/2019/exhibition24/index_en.html
[4]	Video of Danswing Papers, Top 10 finalist of the 2018 Best Illusion of the Year Contest, http://illusionoftheyear.com/2018/10/danswing-papers/