Applied Neuroscience Technology toward Mind-to-mind Communication by Visualizing the State of the Brain to Deepen Mutual Understanding between People

Achieving mind-to-mind communication by visualizing brain waves and conveying accurate information

—Dr. Shimizu, what exactly is “applied neuroscience technology toward mind-to-mind communication?”

“Applied neuroscience technology toward mind-to-mind communication” that I am now researching is a technology that aims to establish communication in which information that one person wants to convey to another person is conveyed with 100% accuracy by using knowledge and technologies in neuroscience. In traditional communication using language, information cannot be conveyed to another person with 100% accuracy due to small misunderstandings (miscommunications) or failure to establish a conversation (discommunication). For example, if a person was to say “Please, something a bit more classy,” the image arising in the other person’s head by the word “classy” depends on the person, which makes it easy for a communication mix-up to occur. To solve this traditional problem, I began research on achieving mind-to-mind communication that can convey information with greater accuracy by visualizing the state of the brain using knowledge from neuroscience. I believe that conveying information to another person with greater accuracy through mind-to-mind communication can lead to new solutions, facilitate consensus building, and maximize the degree of satisfaction with the communication results among the parties concerned.

To give some background to this research, modern society places importance on respecting the various ways of thinking that people have and understanding diversity. Here, the first step in recognizing such diversity is to ask oneself “how do other people think.” I believe that doing so will enable more accurate understanding of another person’s way of thinking, which, in turn, should lead to the creation of new things and new value. By the way, the period in which I began this research coincided with the COVID-19 pandemic when the opportunities for remote communication were plentiful. In everyday communication, it is possible to communicate in a way that includes the atmosphere of the place where you and your conversation partner are present. This is not possible with remote communication, so solving this problem was one short-term goal that I set.

As my ultimate goal, I am carrying out my daily research with the aim of making mind-to-mind communication part of a new communication tool that transcends the telephone, mail, chat, and other traditional means of communication. In short, my goal is to achieve technology that will enable people to understand each other’s sensibilities and the way that others feel and perceive things in a more accurate manner to promote mutual understanding.

—What specific approach do you take in conducting your research?

As a specific research approach to achieving mind-to-mind communication, I constructed “brain-representation visualization technology” for perceiving brain expressions in real time through multidimensional representation. With this technology, we can display geometrical figures based on the state of brain waves during communication making it possible for a person to not only understand the inner state of another person with greater accuracy but to also grasp one’s own emotional state. In my current research, I have succeeded in transmitting a portion of a person’s inner state that up to now could not have been done by extracting only distinctive features from “brain expressions” inferred from brain waves in several hundred dimensions and representing them as low-dimensional information.

Since brain waves arise due to neuron activity in the brain, I hold the hypothesis that “a neuron group should behave with a characteristic pattern according to what that person is currently thinking or feeling.” In my present stage of research, I am undertaking brain-wave measurements under a variety of conditions based on this hypothesis. As a measurement method, I am studying brain-wave measurement using a consumer-oriented electroencephalograph (EEG) based on my personal idea of “wanting to disseminate technology throughout society.” Here, however, I am struggling with the problem that collecting reliable data is difficult due to the noise carried by sources other than brain waves. On the other hand, while EEGs for research applications can be used to collect basic data, this requires the application of gel and electrolytic solution between electrodes and the scalp when measuring brain waves to suppress electrical resistance, so considerable labor and time is needed to collect data in this way.

In addition, human emotions are complex. They can be subdivided into an endless number of emotions with fine gradations making it impossible to express them in only easy-to-understand categories like “happy” or “sad.” I have therefore concentrated on human “discomfort.” This is because I thought that understanding “discomfort” in a variety of scenarios—such as “Did the other person understand me completely?” “Am I stuck on something in my mind?” or “Is this a new discovery?”—could be useful in communication.

As one research result, I discovered that brain waves that indicate N400 and N600 negative potential fluctuations (event-related potentials) occur as a brain response when one’s own way of thinking disagrees with a proposal given by another person. Moreover, in the case of accepting a proposal despite the fact that it is not consistent with one’s own way of thinking, I found that this event-related potential is smaller than that when not accepting the proposal. Research on brain response to such “discomfort” has existed in the past, but that research targeted responses to text such as “the earth is square” or “the color of sunflowers is blue” that would make anyone feel uncomfortable. With this in mind, my present goal in research is to deepen my knowledge while discovering new responses by investigating “whether an uncomfortable response the same as found in prior research would occur when ways of thinking differ between people” and “what kind of response would actually occur if different from that of prior research.”

The initial plan when first launching this research was to extend communication without targeting any specific person. However, as studies progressed, I wondered whether reading the brain waves of amyotrophic lateral sclerosis (ALS) patients could generate facial expressions for conveying their emotions to other people. The jumping-off point for my research here occurred amid much research and development of a brain-machine interface (BMI) that transmits thoughts for inputting text or selecting a command. It was at that time that I wondered whether emotions could be expressed as an aid to transmitting thoughts. Then, as times changed, the remarkable progress in generative artificial intelligence (AI) functions that could generate images of facial expressions provided a strong tailwind to my research. Against this background, the catalyst for initiating my research was the idea that “if information on the kind of facial expression that is being attempted could be read from brain waves, couldn’t generative AI interpret that information and generate human facial expressions?”

In the end, this research also became a study of technology that could be useful in communicating with generative AI. Although generative AI is a powerful tool, it is essential that the user correctly convey his or her intention. Generative AI requires that a command or instruction be conveyed to AI in linguistic form, but what is actually conveyed may differ from one’s original intent. This is a problem that occurs frequently. I therefore would like to use “brain-representation visualization technology” to achieve a world in which conveying brain waves directly to AI will enable everyone to enjoy the benefits of AI regardless of language ability (Fig. 1).

Fig. 1. Overview of brain-expression modeling and perception.

Becoming a researcher who can proudly tell people about one’s own technology

—Please tell us about your research vision going forward.

My research theme is “digitalizing humans by modeling brain information and its processing process,” which is fundamental research for creating digital twins as part of the Innovative Optical and Wireless Network (IOWN) vision put forward by NTT. It is predicted that research on determining the inner state of a person will span a long period of time due to many unknown areas and an abundance of issues. Going forward, I would like to make progress in achieving our major goal of digitalizing humans to enable human information processing by clarifying and modeling brain mechanisms based on neuroscience while investigating engineering methods based on the knowledge gained in that process. If the human brain can be expressed as digital information, it should be possible to not only increase our understanding of the brain but to also use digital models to simulate things about humans that could not be understood in the past since there was never any attempt to do so. For example, it would be possible to determine beforehand “whether work could be made more efficient by forming a team with a certain combination of people and assigning certain roles to each of them.” The field of neuroscience is still developing, and I myself do not yet see a specific path toward the above objective, so I would like to make more progress from here on.

In addition to the above, I would like to research technology that could guide the state of the brain from the outside. The way that things are perceived not only differs from person to person but also changes greatly according to one’s current state. For example, the world may look bright when one is in good health. So my approach here is to search out a method for guiding the state of the brain not by directly stimulating and controlling the brain but by clarifying the mechanism of human sensibility using neuroscience and to use that mechanism to transform the way in which a person perceives things.

—Dr. Shimizu, can you leave us with a message for researchers, students, and business partners?

Yes, of course. The reason why I chose NTT originated in thoughts like “I wanted to develop services that could actually be used in the world” and “I wanted to disseminate in the world services that I could proudly say were technologies that I myself created.” Actually, the origin of these feelings was that I wanted to tell my parents, who were not very familiar with research, about technologies that I created. Now, however, I would like to tell my children when they get older and any future children about my research. This all comes out of a feeling of wanting to proudly tell people about my work and to create technology that can touch all sorts of people in the real world. This view of mine is largely unchanged today.

In the beginning, my research was closer to “applied” in terms of “basic” and “applied” research. So today, it’s extremely interesting as a researcher to wonder whether I can achieve something new by combining the various forms of knowledge that I possess. In the field of neuroscience, I think that there are areas in which I cannot keep up with scientists who have been conducting basic research in this field for many years, even with much study. On the other hand, there are few researchers that have a varied background like mine, so there are pleasurable moments in which I can use my experiences in finding ways of using knowledge gained from fields that I’ve been involved in such as media and signal/image processing.

At the same time, basic research in the field of neuroscience that I am currently engaged in is still far from mature, so while there are things that I certainly want to achieve, there are many things that I cannot due to a lack of knowledge. At present, I am fortunate to be in an environment at NTT in which basic research can be pursued over the long term, so I feel that I can pursue my research steadily and approach my goals over time. Moreover, in basic research, discovering something new that shakes up my intellectual curiosity can be a very interesting moment, and in applied research, receiving feedback from the real world can be very enjoyable, so an environment in which you can experience the blessings of being a researcher is precious.

Since neuroscience is not my original specialty, I make an effort to acquire basic knowledge about neuroscience by listening to lectures given by a variety of people and reading papers and publications. From here on, to be able to say proudly that this is my research, I would like to build up my research achievements from the “basics” that are all so important in research. If any of you reading this article are involved in technology, I would like to encourage you to undertake the research of technology that “you can tell people about without any sense of shame.” In addition, I hope that you continue to have much faith in your research so that you can proudly say “This is correct!” or “This is a new discovery!”