Design a World Where Everyone Can Flourish by Deciphering the Future of Individuals, Society, and the Earth—Communication Science That Connects the Past, Present, and Future through Diverse Knowledge and Technologies

1. Deciphering individuals

In everyday communication, a person can make inferences about how another person is feeling or what they are going to do on the basis of their behavior or situation, even if they do not express words or facial expressions. Many current information and communication technology devices require explicit commands, voice, and gestures, but if these devices can understand the other person’s state of mind like a human, we can expect more natural and smooth communication between humans and machines.

We at NTT Communications Science Laboratories (CS Labs) have been researching mind-reading technology [1], which interprets such latent mental states from unconscious bodily movements and automatic physiological responses. We are conducting research on estimating various cognitive states of people in real time by non-invasively and conveniently measuring eye movements and pupil changes, which is information that appears on the surface of the body. Previous research has shown that the pupil of an observer constricts when they see an attractive face; conversely, when the pupil of an observer constricts unconsciously by changing the luminance contrast around the image of the face, the attractiveness of the face they are looking at increases. This is the first result to show that the constriction of the pupil affects the judgment of preference, while at the same time reversing the conventional theory that the pupil dilates when looking at attractive faces. In another study, we also found that the pupils’ response to the brightness of the visual stimulus presented in the direction of the sound to which we are paying attention reflects which sound we are paying attention to among sounds presented separately to the left and right ears via headphones [2]. This result not only suggests that the brain mechanism of attention is common to some extent between vision and hearing but also the possibility of developing hearing-support devices such as hearing aids by automatically judging and extracting the direction and object of the sound that the user wants to hear from the pupil response.

The aforementioned mental states, i.e., facial preference and attention, are often felt unconsciously, and it is difficult for a person to clearly articulate or become aware of them. An article in this issue introduces the latest research results on mindfulness meditation’s physiological, psychological, and neural mechanisms [3], which has attracted attention as a practical method for cultivating the state of being aware of sensations, emotions, and other experiences in the present moment without suppression.

2. Deciphering our society

Next, I will introduce research cases to understand various social situations caused by various human interactions.

To prevent the spread of COVID-19, new lifestyles such as telework and remote classes are becoming more common. However, fewer opportunities for face-to-face interaction have led to an increase in the number of people who feel isolated, alienated, and anxious with a diminished sense of belonging. To understand and pursue people’s future well-being, that is, the state of being physically, mentally, and socially fulfilled and alive, it is important to clarify the situations and factors that improve the well-being of individuals. To clarify this issue from the perspective of the relationship between individuals and teams and society, CS Labs developed a scale to measure personality traits of people toward teams and society based on the view of “Self as We” advocated by Professor Yasuo Deguchi of the Graduate School of Literature at Kyoto University. We also developed a method and smartphone application to measure the well-being of individuals and teams in a multifaceted and sustained manner, which was released in 2020 [4]. To capture changes in an individual’s subjective mental and physical states, the smartphone application records intuitive and physical experiences as embodied emotional expressions (EEEs), which uses exclamation words, onomatopoeia, and pictograms, and these records are visualized in two dimensions using the embodied emotional expression map (Horizontal axis: pleasant/unpleasant; Vertical axis: high/low arousal) (Fig. 1). This makes it possible to capture subjective and ever-changing mental states before and after various experiences while reducing the burden on the user. This can also be used by members of a group who are having a shared experience to analyze the relationship between group well-being and mood changes such as meeting satisfaction.

Fig. 1. Intuitive recording and visualization of mental and physical states using EEEs.

We also developed a method for visualizing and sharing values about what improves the well-being of individuals. We analyzed 3900 responses to a questionnaire survey from about 1300 people asking them to list 3 things (values) that are important to them. The results indicated that those values can be classified into 4 categories: self-related (I), relationships with others (We), relationships with local communities and society (Society), and relationships with the world and nature (Universe). Our Well-being Cards (Fig. 2), which are based on this classification of values, are used in various workshops for elementary school students as a tool to deepen their understanding of their values and the diversity of the values of those around them by sharing them with each other. Our Well-being Cards are available for download from the web with instructions on how to use them [5].

Fig. 2. Our Well-being Cards.

We have seen a lot of news and articles about ChatGPT, which is a generative artificial intelligence (AI). At CS Labs, we are conducting somewhat unusual research using interactive AI technology. We developed a dialogue agent that helps users who have difficulty in directly communicating with others, such as patients with depression, to disclose their problems and communicate them to experts [6]. After establishing a trusting relationship through a conversation with a chatbot that is an interactive agent and allowing both parties to self-disclose, we conducted an experiment to determine how the trust relationship between users and experts changes when the chatbot introduces a trusted human expert.

In the experiment, 47 participants chatted with the chatbot for about 15 minutes every day for 4 weeks. The experimenter interviewed them online for 1 hour. The participants who only built trust with the chatbot as Step 1 and those who were later introduced to an expert by the chatbot as Step 2 were asked to rate their trust in the chatbot and the expert on a 7-point scale (1: not at all trustworthy to 7: very trustworthy). There was no change in the level of trust in the chatbot between the cases in which Step 2 was not conducted and those in which Step 2 was conducted. The level of trust in the chatbot was as high as 6, while that in the expert was significantly higher when Step 2 was conducted. We also found that when the chatbot asked the participants if they could share what they had disclosed to it with the expert, they were more likely to be positive and motivated by it. Most research on interactive agents had focused on the relationship between people and agents, but this research is unique in that it aims at connecting people through agents. Technology that learns the relationship between people from conversations and situations will become increasingly important, and CS Labs is conducting research that uses AI to recognize the purpose of dialogue and dialogue situations from multimodal information such as audio, image, and language information in dialogue scenes [7].

3. Deciphering the Earth (universe)

It is the ultimate goal of science to clarify and reproduce the structure and laws behind all things in nature. Advances in sensing technology have led to the acquisition and distribution of high-definition and high-quality image data, for example, observations of typhoon development by the Geostationary Metrological Satellite Himawari. With the increasing number of natural disasters, expectations for predicting the future of complex physical phenomena related to fluids, such as weather and ocean currents, are increasing by accurately modeling their development from observation data and reproducing them through simulation. Humans have manually discovered the laws behind physical phenomena (e.g., Newton’s laws of motion, equations of motion, etc.) from observed data. However, the more complex the phenomenon, the more difficult it is to model, and the more limited it is to design equations manually. The discovery of such laws and the automatic simulations that faithfully reproduce phenomena using AI is a very challenging task. In an article in this issue, the research of physical simulation based on the data-driven approach is introduced in detail as a machine-learning technology that reproduces physical phenomena only from observation data without manually designing equations in advance [8].

The above is an example of research aimed at modeling physical phenomena. For complex analyses such as meteorological data, it is necessary to conduct enormous calculations at high speed. There are increasing expectations for quantum computers, which are increasingly being reported for practical use. However, overcoming quantum errors is an extremely important issue to resolve to put such computers into practical use. Quantum computers can execute high-speed parallel calculations using quantum bits, which is quantum mechanical superposition information. However, since errors due to noise are likely to occur in quantum calculations, it is important to have a technique to verify whether the calculation results of a quantum computer are correct. An article in this issue describes methods for efficiently verifying the correctness of quantum-computer calculation results [9].

In terms of understanding the universe, the Greek philosopher and mathematician Pythagoras preached around the 6th century BCE that “All Is Number” and believed that everything in the universe could be understood by mathematics according to the laws of numbers rather than human subjectivity. From the latest research results of the Institute of Fundamental Mathematics established in CS Labs in October 2021, we developed a method for calculating the heat kernel, which is the basic solution for the time evolution of a mathematical model that describes the minute interaction between light and matter, such as atoms, using an algebraic theory called representation theory [10]. This study also reveals an interesting relationship between the heat kernel and zeta function in the Riemann hypothesis, a well-known open problem in modern mathematics.

4. Designing our future

In this article, I have introduced research efforts at CS Labs from the viewpoint of deciphering individuals, society, and the Earth (universe). In addition to AI, technological progress will continue to accelerate, but we are also being asked about our responsibility and ethics in creating and using technology. We will deepen our understanding of the diversity of individuals, societies, and the Earth, which is constantly changing, and pursue the truth of the universe. We will also design a sustainable and better future society in harmony with individuals, societies, and the changing global environment and conduct further research and development toward this goal.

References

[1]	M. Kashino, M. Yoneya, H.-I. Liao, and S. Furukawa, “Reading the Implicit Mind from the Body,” NTT Technical Review, Vol. 12, No. 11, Nov. 2014. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201411fa6.html
[2]	H.-I. Liao, H. Fujihira, S. Yamagishi, Y.-H. Yang, and S. Furukawa, “Seeing an Auditory Object: Pupillary light Response Reflects Covert Attention to Auditory Space and Object,” Journal of Cognitive Neuroscience, Vol. 35, No. 2, pp. 276–290, Feb. 2023. https://doi.org/10.1162/jocn_a_01935
[3]	M. Fujino, “Understanding Mindful Awareness in Mindfulness Meditation—Investigation of the Psychological, Physiological, and Neural Mechanisms of Mindfulness Meditation,” NTT Technical Review, Vol. 21, No. 10, pp. 20–25, Oct. 2023. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr202310fa3.html
[4]	N. Saijo, M. Fujino, A. Murata, Y. Ooishi, and J. Watanabe, “Human Information Science Research for Understanding and Improving Our Well-being,” NTT Technical Review, Vol. 20, No. 10, pp. 37–43, Oct. 2022. https://doi.org/10.53829/ntr202210fa5
[5]	Website for downloading Our Well-being Cards (in Japanese), https://socialwellbeing.ilab.ntt.co.jp/tool_measure_wellbeingcard.html
[6]	Y. Lee, N. Yamashita, and Y. Huang, “Designing a Chatbot as a Mediator for Promoting Deep Self-Disclosure to a Real Mental Health Professional,” Proc. of ACM Human-Computer Interaction, Vol. 4, No. CSCW1, Article no. 31, pp. 1–27, 2020. https://doi.org/10.1145/3392836
[7]	Y. Chiba and R. Higashinaka, “Analyzing Variations of Everyday Japanese Conversations Based on Semantic Labels of Functional Expressions,” ACM TALLIP, Vol. 22, No. 2, Article no. 52, pp. 1–26, 2022. https://doi.org/10.1145/3552310
[8]	Y. Tanaka, “Machine Learning That Reproduces Physical Phenomena from Data,” NTT Technical Review, Vol. 21, No. 10, pp. 15–19, Oct. 2023. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr202310fa2.html
[9]	Y. Takeuchi and S. Tani, “Dilemma between Quantum Speedup and Computational Reliability—Overcoming Errors with Efficient Verification Methods for Quantum Computing,” NTT Technical Review, Vol. 21, No. 10, pp. 26–29, Oct. 2023. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr202310fa4.html
[10]	C. Reyes-Bustos and M. Wakayama, “Heat Kernel for the Quantum Rabi Model,” Adv. Theor. Math. Phys., Vol. 26, No. 5, pp. 1347–1447, 2022. https://dx.doi.org/10.4310/ATMP.2022.v26.n5.a8