Be Insensitive to Peer Pressure to Fight a Fierce Battle of Ideas

Leading the world in research on Shitsukan perception

—Please tell us about the research you are currently working on.

I’m researching the human perception of specific properties of objects called Shitsukan (Japanese word for the sense of quality). Humans recognize various Shitsukan through their five senses, including vision, and instantly judge physical properties, material properties, conditions, and subjective value. Their ability to recognize Shitsukan plays an important role in human activity because humans are involved in the environment through recognizing and evaluating objects, making decisions, and controlling their body actions, all of which are deeply connected to Shitsukan.

The Shitsukan that I’m studying can be roughly divided into two types: (i) physical Shitsukan, such as physical reflectance properties (e.g., glossiness and transparency), materials (e.g., ceramics, metals), and conditions (e.g., dry and frozen), and (ii) subjective Shitsukan, such as beauty and preference. I believe that Shitsukan is the ability of the human brain to decode the nature of an object from sensory input. Many mechanisms with which the human brain recognizes the diverse Shitsukan of objects in the real world remain unknown. Explaining these mechanisms is critical not only for scientific understanding of human-sensory-information processing but also advancements in information-engineering technology for recognizing and generating the Shitsukan of real objects.

I’m one of pioneers of Shitsukan research. We started studying human material perception in the mid-1990s and made a few important contributions, including discovery of low-level image features of surface gloss, by 2010. Since then, using a joint-research grant called “Grant-in-Aid for Scientific Research on Innovative Areas” from the Ministry of Education, Culture, Sports, Science and Technology (MEXT), we have been promoting world-leading interdisciplinary research on Shitsukan. From fiscal year 2010 to 2014, I headed a research team on the theme “Human Shitsukan perception from visual, auditory and tactile information” in the interdisciplinary area research project “Brain and Information Science on Shitsukan (BISS) (Integrative studies of neural mechanisms and advanced information technologies for perception of material and surface qualities)” [1]. During that period, we studied color-luminance interaction related to gloss perception and clarified the mechanism for recognizing the Shitsukan (viscosity) of a liquid on the basis of motion information. By exploiting these human mechanisms, we developed a light-projection-mapping technology called Hengento, which creates the illusion that a still object appears to be moving.

From fiscal year 2015 to 2019, I headed the second interdisciplinary area research project “Innovative Shitsukan Science Technology (ISST) (Understanding human recognition of material properties for innovation in Shitsukan science and technology)” [2], and we pursued Shitsukan research from the perspectives of information science, neuroscience, and psychophysics and established the academic field of Shitsukan science and technology, which ranges from basic research to applied research in the fields of vision, touch, hearing, and language.

As a principal investigator of the research team “Visual, auditory and tactile Shitsukan recognition mechanisms based on signal modulations” within the above project, our team clarified the perception of (i) wetness on the surface of an object on the basis of luminance and color statistics [3] (Fig. 1) and (ii) ultrafine structures on the basis of a reduction in image contrast. At the same time, we analyzed the mechanism of perception of liquid texture by using an artificial neural network [4] (Fig. 2).

Fig. 1. Transforming image texture to make its surface look wet [3].

Fig. 2. Computer-simulated images used to analyze the perception of liquid Shitsukan (viscosity) [4].

Extending the principle of Hengento, we developed a method of synthesizing ghost-free stereoscopic images called Hidden Stereo that does not cause blur when an image is viewed without stereo glasses [5]. The field of Shitsukan science and technology that we established through these two interdisciplinary area research projects (BISS and ISST) has been highly evaluated as internationally outstanding in terms of interdisciplinarity and breadth of vision.

—You have established a new academic field concerning Shitsukan. What is your research going forward?

I’m interested in human information processing, in particular, visual information processing. When considering politics and society, for example, I’m also interested in the reasons behind the actions of people. I’m always trying to understand humans not only using established disciplines such as psychology but also looking for novel approaches that make sense to me.

In 2020, we started the third Shitsukan project, Transformative Research Areas research project “Deep Shitsukan (Analysis and synthesis of Deep Shitsukan information in the real world)” [6]. I again acted as head of the project. The key concept of the project is that to understand the essence of Shitsukan, it is necessary to understand (i) the superficial Shitsukan information processing that links the input information captured by the sensory organs to Shitsukan-attribute variables and linguistic labels of Shitsukan categories and (ii) the processing hierarchy that lies behind superficial Shitsukan, which we call Deep Shitsukan.

We currently assume four types of Deep-Shitsukan processes. The first process is calculating the multifaceted ecological meaning and value of things and events from Shitsukan information and often inducing an emotional response inside the body. The second process is predicting the outcome of behaviors and selecting the appropriate behavior by constructing an external model in the brain by integrating Shitsukan and other sensory attributes. The third process is determining how Shitsukan-information processing is influenced by the characteristics of the person doing the processing (for example, age, brain dysfunction, cultural background, and experience). The fourth process is discriminating between real and fake through the sensory information on real objects acquired from the five senses.

In this project, we aim to elucidate human Deep-Shitsukan processing from the perspective of cognitive neuroscience and develop innovative Deep-Shitsu­kan technology from the perspective of engineering. We will reveal the essence of sensory information that allows people to experience realistic Deep Shitsukan. We will also develop machine-vision technology for recognizing Deep Shitsukan and media technology for manipulating it at will, with which we expect to connect Shitsukan science to art. Our project consists of ten teams, each belonging to one of three research themes (Fig. 3). As the leader of one of the teams, I’ll focus on a unified understanding of Deep-Shitsukan recognition from multiple angles, namely, vision, hearing, touch, and linguistic information, using psychophysics and sensory engineering.

Fig. 3. Deep Shitsukan research project.

A paradigm shift occurred as a result of machines catching up with humans

—We talked to you a while back in 2012. How has your research environment changed during that time?

Around 2012, human capabilities were still ahead of machines in a variety of intelligence-related tasks. However, over the past eight years, as artificial intelligence (AI) and machine learning advanced significantly, machines have been able to surpass humans, albeit to a limited extent, in terms of intellectual capabilities. This paradigm shift has had a considerable impact on me as a researcher of humans. By understanding the complex mechanisms of human cognition and behavior, we have been trying to obtain clues for developing machines with abilities comparable to humans. However, the exact opposite approach is now becoming commonplace. That is, attempting to understand human information processing by analyzing machines that have acquired human abilities by machine learning. This research revolution is taking place in neuroscience, and I want to fully use this new approach for advancing our research on Deep Shitsukan.

As these technologies and research are rapidly advancing, I’m studying new technologies with young researchers. A huge number of papers are published on the Internet, and it is difficult to choose which to read. I have a lot of work other than research on my hands, so it’s not easy to find time to catch up on these advances. Although I’m excited about the new waves of discoveries that are coming one after another, I feel it difficult to stay on the cutting edge at all times. Therefore, I am trying to create my originality by fusing the newly acquired knowledge with the experience and knowledge I have cultivated thus far. I saw in the past that the basis of research on human senses had shifted from psychology to imaging of brain functions using functional magnetic resonance imaging, and I now see the field is moving toward fusion with AI research. I believe that having such a long research history will be an advantage over young researchers who only know the latest state-of-the-art approaches.

I think these are the fastest changes in research trends I’ve ever seen. I’ve heard from doctoral students who research machine learning and AI that the research they were working on in their first year had become outdated by their third year and no longer worth publishing. Today, they have rivals around the world, and when they publish their research results, another researcher will publish results superseding theirs within a year. They say that researchers’ minds cannot rest. I think that researchers in AI are living in an exciting time but face fierce battles and a tough time mentally.

—What is the driving force behind consistent research activities in research areas that are showing tremendous development?

That’s a difficult question. I tend to jump on anything new. Even though it may seem like I am sticking to one topic, my interests are constantly changing and evolving. This is my research attitude. People tend to paint a picture of researchers as pursuing a single theme, but I think I’m a little different in that I’m consistent with a sense of wanting to do something interesting.

Since the subject of my research, i.e., Shitsukan, is a very difficult problem, I have launched several projects on this topic to force me to work on it, so it may appear that I am sticking to one theme. However, I always enjoy listening to new ideas from many researchers participating in the project. Some ideas are very stimulating and interesting enough for me to change my research.

I don’t accept popular research trends, even if it’s a subject of interest, unless I really believe they are right. On the other hand, if I find something interesting, I want to adopt it even if it’s new and unfamiliar to me. Although my interests vary, there are several reasons that I have remained a researcher. The first is because researchers don’t take orders from others. Since my youth, I’ve been reluctant to follow instructions from others. I’m grateful to my bosses for allowing me to keep doing so. Fortunately for me, NTT Communication Science Laboratories has been a company that respects researchers and lets them do what they want to do. I think that if such an environment disappears, it would become less attractive to researchers.

Young researchers, your time has come!

—Researchers also need to select a research environment to gain that freedom.

I think that success in research results from a series of coincidences. To get the most out of such coincidences, it is important to be able to move freely. As long as you receive research funds, you need to make plans and conduct research in accordance with the plan; however, as many researchers including Nobel laureates have said, it is important to leave such planning to researchers to a certain extent, give them more freedom and encourage them even if only 1% of them succeed and 99% fail. If we don’t do that, we won’t be able to open up routes to the future, and those failures, or “waste,” are necessary to open the future. It is critical to be able to tolerate this waste to achieve good research results.

NTT has the strength to ensure such freedom. I am also teaching at a university. University faculty members have a tough time obtaining research funding, but NTT researchers do not have such a problem. Researchers may have unusual personalities, even a bit weird. That’s why they can create something different. If that is the case, I think we can open up the path to the future by creating an environment that allows researchers to pursue research freely.

Such a free research environment is becoming ever less common in Japan, and the number of researchers is decreasing as well. Young researchers are having a difficult time trying to obtain one of the few research posts available. In the history of research activities in Japan, corporate laboratories have supported not only applied research but also basic research. Among those laboratories, NTT, with its mission and culture inherited from when it was a public corporation, still has an environment in which researchers can take their time and focus on basic research. I think the role played by NTT laboratories in research activities in Japan is significant.

I am also a member of the Science Council of Japan (Section I: Humanities and Social Sciences), and from that standpoint, I think that Japan should take a strategic approach to nurture the young generation of researchers that will create value in the future. In that regard, NTT has the ability to recruit and train many outstanding young researchers and give them the freedom and discretion to maintain their motivation. I hope that NTT will continue to demonstrate its power and contribute to the future of Japan.

—Please give a few words to our young researchers or students who want to become researchers.

As a humanities graduate with a science background, a corporate researcher, and a university faculty member, I have gained experience in looking at and thinking about things from multiple perspectives. I want to make use of this advantage to train young researchers while actively speaking my mind about the development of human resources at various opportunities.

I think that since the standpoints of young researchers at NTT laboratories and students differ, the method of guidance should also differ. Many NTT’s researchers have earned their doctorate degrees and are already equipped and ready to be researchers, so I encourage them to improve themselves through research. On the contrary, students have not yet decided whether they will become researchers. Accordingly, I do not strongly encourage them to become researchers; instead, I give them research guidance with the hope that it will be useful to them in some way.

With that in mind, my message to young researchers and students is to become insensitive to peer pressure. If we consider the trend of researchers starting their own businesses, as in the USA, as one effect of peer pressure, it may be that peer pressure works well to raise the standard of research. In Japan, however, children are brought up to “be good girls or boys” or “act like everyone else” until they go to university, and I think a strong sense of self-regulation has developed as a result. I want you to discard that self-regulation and become insensitive to such peer pressure in regard to your research activities. In other words, it might be a good idea for you—as a researcher—to dare to be “the nail that sticks out.”

It is an exciting time when the presentations of papers at international conferences on cutting-edge research are immediately published on the Internet, and new methods and technologies come out one after another. In such an era, researchers, especially those in the information field, may find it difficult to reap the benefits of belonging to a large research institute. Researchers in such fields do not necessarily require large-scale experimental equipment and technology in large laboratories for their research, and it has become possible for researchers outside major institutes to enter the ring of competition if they have an idea and a suitable computing environment. In other words, you are living in a time when you have to fight a fierce battle of ideas day and night. New ideas and directions can only come from research sites that are in constant contact with cutting-edge research. You can’t expect your supervisors, who are busy with common tasks, to give you guidance. Therefore, young researchers, your time has come!

References

[1]	Website of the Brain and Information Science on Shitsukan (material perception) project, http://shitsukan.jp/BISS/content/137
[2]	Website of the Innovative Shitsukan Science Technology project, http://shitsukan.jp/ISST/en/outline/index.html
[3]	M. Sawayama, E. H. Adelson, and S. Nishida, “Visual Wetness Perception Based on Image Color Statistics,” Journal of Vision, Vol. 17, No. 5, pp. 7–24, 2017.
[4]	J. J. R. van Assen, S. Nishida, and R. W. Fleming, “Visual Perception of Liquids: Insights from Deep Neural Networks,” PLoS Computational Biology, Vol. 16, No. 8, e1008018-29, 2020.
[5]	T. Fukiage, T. Kawabe, and S. Nishida, “Synthesizing Ghost-free Stereoscopic Images for Viewers without 3D Glasses,” NTT Technical Review, Vol. 15, No. 11, 2017. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201711fa5.html
[6]	Website of the Deep Shitsukan project (in Japanese), http://shitsukan.jp/deep/