Elucidating the Implicit Brain Functions that Enable Flexible Human Behavior

Unravelling implicit brain functions

—First of all, could you tell us a little more about your research?

The research that has been consuming most of my time recently has been focused on sports brain science, which aims to clarify an athlete’s brain function and improve his or her performance. I started to work on this research in earnest when the Sports Brain Science Project [1] was officially launched in January 2017. Exercise physiology and biomechanics were (and still are) the mainstream subjects, so our research was a peculiar addition to the field of sports research. My area of specialty was originally brain mechanisms such as hearing that are involved in perception and cognition [2], and I am continuing that research now. Several years ago, developmental disorders such as autism spectrum disorder (ASD) were also included in the study [3].

Although it may seem like I was putting up my hand in a very different field, within myself, I intended to base my research on a consistent awareness of issues. In general, humans can act extremely flexibly and adaptively in various environments and situations, and it is problem awareness that makes that ability possible. Recent progress in artificial intelligence (AI) has been remarkable; however, at present, it can be said that it only works for limited targets or situations. For example, within the specific rules of the board game go, a system that can beat a go master is completely useless elsewhere.

In contrast, people make a huge number of decisions and carry out various activities from the moment they wake up to the moment they go to bed. Even brushing one’s teeth with the appropriate pressure, running to the train station while avoiding cars and other people, and talking in noisy bars are actually extremely difficult when viewed as information-processing tasks. Nevertheless, the person performing those tasks doesn’t feel as if he or she were doing anything difficult. That is, we are totally unaware, and we cannot explain why we can do such things. It is often difficult for AI to do things that human beings do unconsciously. So how does the brain enable a person to work on a very difficult information-processing task without trying hard? This is the problem awareness that prevails in each research theme.

One of the most sophisticated forms of such flexible human information processing is the performance of top athletes. In martial arts and ball games, the state of the bout or game changes rapidly, and while that’s happening, it’s necessary to achieve—in an instant—the optimum physical movement according to the situation on the spot, the behaviors of the opponent and the ball, and so on. Given the variety of situations, the required accuracy, and the shortness of time allowed, it is easy to imagine how difficult performing those tasks is. Top athletes achieve high performance as a result of solving those difficult tasks; it therefore follows that if we study their brain functions, we should reveal a concrete algorithm.

At the same time, the findings of such a study may also be useful to the athlete. The higher the level of the match or bout gets, the less likely it is that the result will be determined by physical ability alone. For example, in the case of professional baseball, there are some first-string players considered to be ace level even though their pitching speed is not very fast, while other players may get stuck with the second-string players even though they throw a good fast ball. Physical training alone is not enough to fill in this gap between these types of players, and cognitive training is essential. Establishing these methodologies specifically and systematically is one of the goals of our research on sports brain science.

Meanwhile, in contrast to top athletes, some people cannot easily do certain tasks that many people think can be done effortlessly. One group of such people consists of those with the developmental disorder ASD. ASD is caused by peculiarities of the innate brain function. It is characterized by the fact that a person with ASD cannot communicate with others well or that his or her interests and activities are significantly biased. The so-called “inability to read the situation,” that is, the inability to speak or act according to the situation, or to panic in an unfamiliar situation, makes everyday life difficult for people with ASD. This condition may be related to the implicit brain function that adapts flexibly to various situations. Some ASD sufferers are very intelligent and may exert extraordinary abilities in certain areas. However, it is difficult for many of them to do things that other people can do roughly or unconsciously.

In some way, this state of affairs may be partly reminiscent to the current state of AI. If we study such an example, we will be able to clarify—from a different angle to that of the case of top athletes—the principle of achieving flexible behavior according to the situation. Of particular interest are the characteristics of the human sensory and motor systems. Some ASD sufferers are often troubled by hypersensitivity, such as feeling strong discomfort in the presence of certain sounds, or having the ability to hear sounds normally in a hearing test but not being able to hear a conversation in their daily environment.

Moreover, some of them show movement characteristics such as clumsy hands or very awkward movements. These facts indicate the possibility that the “body,” namely, a device that takes in information from the outside world and outputs actions back to the outside world, plays an essential role in flexibly adapting to the situation. In this respect, research on ASD is linked to that on athletes. I think that linking research on those topics from now on will give us some hints for expanding the scope of the application of AI.

Understanding individuality and valuing diversity

—What kind of findings have been achieved in the research on sports brain science over the last two years?

I am currently conducting research focusing on baseball and softball. Thanks to the cooperation of professional baseball teams from Japan and the United States, semi-professional baseball teams, university baseball teams, the Japan Softball Association (Japan Women’s National Team), and teams from the Japan Women’s Softball League, we have gathered a considerable amount of data on top-level players (Fig. 1). Many parts of the data cannot be announced yet owing to contractual and other reasons, but the results of this cooperation are pretty exciting because the difference between top-level players and players that do not reach that level is clearly visible to a greater extent than we expected.

Fig. 1. Image of the smart bullpen.

As an example, I’ll explain some research on batting in baseball and softball. In an actual game, the pitcher skillfully throws balls with various trajectories so that the batter cannot easily hit the ball. Unlike when a batter practices swings and hits with a batting tee, where it is usually only necessary to produce a fixed swing, hitting actual pitched balls requires a very flexible response. To reveal the details of this response, we conducted an experiment in which the pitcher randomly threw two kinds of balls at different ball speeds to the batter. The batter was instructed to hit the ball if the pitched ball was a strike (in the strike zone) and not to hit it if it was a ball (outside the strike zone).

The participants were women’s softball players including ones representing Japan. The participants wore wearable sensors, enabling us to obtain data from their movements. Analysis of the batters’ body movements clearly distinguished the batters who were able to adjust the timing of their bat swing in response to the speed of the ball from batters who could not do so (Fig. 2). It was estimated from the analysis results that in the former case, the timing of the batter’s swing is determined on the basis of visual information (i.e., pitching form before and after release of the ball or the trajectory of the ball immediately after release) that was obtained about 0.1 seconds after the ball was released.

Fig. 2. Example of feature analysis of the physical movement of a batter.

To further specify the visual information used for judgment, we used virtual reality (VR) to create a condition in which the information concerning the actually measured pitching form and that concerning ball trajectory were interchanged (as if a slowball (changeup) was pitched by a pitcher in the form of a fastball or vice versa), and after that informational interchange, we tried to measure the batter’s behavior in the manner described.

Batters who were able to cope with the combination of information concerning the original pitching form and ball trajectory by varying their batting tempo became unable to cope when the combination of pitching form and ball trajectory was interchanged. This result is clear evidence that the batter who can respond at will to the ball uses the pitching-form information to predict the speed of the ball.

Even more interesting to us was the discovery that those batters were completely unaware of the fact that they were using information concerning pitching form. One of Japan’s top batters said, “With this pitcher, I can’t determine the type of ball from the pitching form.” However, contrary to her words, the experimental result clearly showed that she was unconsciously aware of the difference in pitching form.

In this way, the true worth of our research is to reveal what people cannot consciously notice about themselves by using objective data. Consequently, we made full use of various kinds of information and communication technology such as wearable sensors, computer vision, biometric signal processing, machine learning, and VR and tried to estimate the state of the brain or mind from information that could be observed from the outside. The subjectivity of a person is important, but it is often the case that only having subjective information is not enough to reach the truth.

Through this kind of analysis, it has become possible to objectively understand what people have said in the past, for example, from the player’s viewpoint, that something can be done somehow, and the body moves unconsciously, and from the coach’s viewpoint, that a player has talent. It was also found that the performance of players in the above experiment, the index of coordination ability obtained in the experiment, and the batting averages in official games corresponded well. Furthermore, when we tried the same experiment with candidates for the under-14 (14 years old and under) Japanese national team, we learned that at that age, some players can already respond to different ball speeds, and some cannot. This means that the path to evaluating players and discovering talent based on objective data is being opened up.

—It feels harsh to be declared “talentless” at a younger age.

That is often said. However, I think it’s rather the opposite. That is to say, whether a player is superior or inferior is persistently evaluated on a certain axis, and in fact, there are many such evaluation axes for athletes. It is important to know exactly what profile a player has in relation to many evaluation axes. There is more than one ideal type of batter.

According to the player’s profile, the player can try to maximize their strengths and overcome their weaknesses. For example, a certain women’s softball player was very successful as a long-distance hitter right up through high school; however, she has not yet become a regular in the Japan Women’s Softball League. She boasts the best swing speed of the team when hitting at a batting tee. However, in a mixed-ball (slowballs and fastballs) experiment, her average hitting speed was in the slow category. In other words, if she faces a good pitcher, she will not be able to swing fast enough to hit the ball. Analysis revealed that she was unable to determine the type of ball from visual information from the start. That is, she can try to determine the type of ball, but no matter how much she practices swinging the bat, the practice will not be effective.

Alternatively, as another approach, it might be better to give up on unconsciously determining the type of ball and instead try to improve the ability to read the game situation and swing only at certain types of pitches. She shouldn’t worry if her reading is off and she swings and misses. As far as the pitcher is concerned, it is far worse to give up a home run. In fact, even some of the best players are ones who swing only at certain types of pitches.

The challenge for coaches is to provide uniform instruction while ignoring individual profiles. If the player doesn’t train according to his or her aptitude, the training might be ineffective and even worse, it might ruin the player’s potential. However, under the current state of affairs, the profiles of individual players at the top level are not accurately understood, and it seems that coaches often give subjective guidance based on their past playing experience. If coaches and players play differently, or the type of brain processing behind the way they play differs, they may not share images in their minds and may not even understand what each other is saying. After all, even a good player doesn’t really know why he or she can play the way they do. Good use of objective data will help fill those gaps.

Research on sports and ASD is, ultimately, a study of human diversity. The more I study, the more I realize that there are many types and subtypes of physical and cognitive aspects. In contrast, from the traditional viewpoint of human beings, which also applies to basic research in brain science, there has been an assumption of a “standard” human being, with individual differences considered as variations from that standard. That is, there was an implicit premise for understanding individual differences as the mean and variance of normal distributions. This way of thinking tends to lead to valuing the majority or judging value on a particular axis.

However, even the same characteristics can have advantages and disadvantages depending on the goal and environment. Although the idea of curbing individual diversity and approaching a “standard” (i.e., the majority) is becoming prevalent everywhere, it needs to be given caution. In any field, having a multidimensional and objective understanding of characteristics is the starting point of understanding the difficulties faced by the minority and enabling them to maximize their individuality. Once the starting point and goal to be achieved by a person are defined, he or she should be able to see a way to solve a problem. I hope our research will be able to help people solve problems and achieve their goals in any situation.

—What was the trigger for your journey as a researcher?

My father taught organic chemistry at a university, and when I was in primary school, I lived in an official residence on the university campus, so the daily life I observed was the life of a researcher. I chose to be a researcher as a profession without any particular reason; I just thought that’s the way it was. To be honest, I just couldn’t picture myself working in a company, doing business with customers, or any other profession. I’ve been interested in a wide range of topics such as insects, astronomy, architecture, and archeology from an early age, and I get the feeling that I arrived here by being guided by my interests at the time.

I am interested in hearing because for as long as I can remember, listening to sounds has aroused strong emotions in me. For example, when I was little, I picked up vacuum tubes from discarded televisions and stereos and threw them onto the concrete surface, and when they shattered, I felt unbelievable pleasure from the sound they made. In contrast, I felt tremendous fear when I hit the bass keys on a piano keyboard. Moreover, I was very fond of Japanese popular songs for adult audiences, and listening to certain songs made me shiver inside. I used to wonder if other people felt the same way when they listened to music.

In elementary school and junior high school, I made my own radio and audio equipment to listen to music. I wasn’t taught by anyone, and the parts I used came from broken televisions and other appliances. Gradually, through trial and error, I naturally learned the basics of experimenting and measurement as well as the theories of electricity and sound. I think that my attitude rather than the knowledge that I acquired through that process is the basis of my research up until now.

Some of my favorite reads at that time were the Japanese and Chinese classics. In particular, I read essays such as “Tsurezuregusa” and “Hojoki,” military tales such as “The Tale of the Heike” and “Taiheiki,” as well as those from “The Hundred Schools of Thought,” and it was the ancient Chinese philosopher Zhuang Zhou who particularly affected me. On reading many fables such as “The Butterfly Dream” and “The Death of Chaos,” I was reminded of the limit of human recognition as well as epistemological problem awareness, such as the concept that the world in front of our eyes is not actually as steadfast as we naturally think. It is dangerous to believe in a single viewpoint and set of values, and yet it is impossible to completely recognize all viewpoints and values. So what should we do? Such questions are always on my mind.

When deciding my major in the third year of university, I chose the field of hearing because that field is where my interest in sound, epistemological problem awareness, and methodology of experiments intersected. However, that does not mean that professors who specialized in hearing were teaching at the university. Although I learned the basics from professors in fields such as vision and neurophysiology, and also from mathematical models, I had no choice but to manage on my own in regard to what and how to study hearing.

Nevertheless, everything I learnt originally came in a trial-and-error manner in my own way, and I didn’t really like being lectured to very much, so I considered this environment rather cozy. That being the case, I think I took many detours. If I had selected my area of expertise in a laboratory with a lot of talented leaders from the beginning, I could have efficiently studied core themes of the research community by using advanced methodologies, and I would have published more papers. Since that was not the case, it might have made me somewhat unique. Even in everyday life, I rarely enter a shop if there is a line of people waiting outside; that is my personal nature, which is both good and bad.

Follow one’s own interests and nature’s providence, and do not overstate the social value of research.

—You have focused on your interests and unique character. Finally, please give us your message for researchers.

People around me often find me amazed when things such as life’s most important matters are easily taken for granted. In the same manner, I changed my major to another research field; accordingly, the methodology and knowledge that I had cultivated up to that point and the results and personal connections I had accumulated in the research community often got reset. I’m often asked how I can take such risks (like losing what I cultivated by changing majors) or am told I’m very decisive. To be honest, I’m not thinking like that at all. It is more accurate to say that I cannot think like that. I lack the ability to worry about the future, and I only focus on my current interest and problem awareness. I think people would be unhappy to end up doing things that they are not interested in as a result of thinking too much.

I cannot say that I’m able to read every situation or other people’s feelings well. Because I belong to an organization, I should probably compromise and accept situations where I’m not completely satisfied; however, I am lucky to be in a work environment in which I was able from the beginning to get involved in basic research and my approach was accepted. If a standard for judging the “goodness” or “badness” of what you are doing exists, the only question is whether it suits the providence of nature. Although judging the value of a certain matter may change according to the situation and standpoint, being suitable for natural reasons remains until the end. I think that basic research is that which does not go against the laws of nature.

As long as you are engaged in work for which you receive research expenses and are provided with a research environment, human resources, etc. (in other words, you are not just pursuing a personal hobby), it will naturally be necessary to explain what kind of value that work brings to society. When I explain that value to society, what I most care about is not to overstate the value. That is different from fabrication of data, which is also bad.

Overstating the value means, for example, in the introduction of research proposals, etc., writing “This is useful for such things” without proper grounds. There are some papers written with statements like “for the elderly” or “for disabled persons,” without knowing the real problems faced by those people and there are also papers that have a huge leap between what is said to be useful and specific research contents. I try to listen to the words of athletes and ASD sufferers as much as possible so as not to exaggerate my research’s value. That is why we have narrowed the scope of our research to some extent. By listening to people carefully, we will find real problems and find ways to dig deep and solve them as basic research. Because our research can affect the lives of athletes and ASD sufferers, the above-mentioned process or the results achieved by the process cannot be neglected. And if research findings are truly valuable, people will recognize that, and business opportunities will be created accordingly.

References

[1]	Sports Brain Science Project, http://sports-brain.ilab.ntt.co.jp/index_en.html
[2]	M. Kashino and H. M. Kondo, “Functional Brain Networks Underlying Perceptual Switching: Auditory Streaming and Verbal Transformations,” Philos. Trans. R. Soc. Lond. B Biol. Sci., Vol. 367, No. 1591, pp. 977–987, 2012. doi: 10.1098/rstb.2011.0370.
[3]	I. F. Lin, A. Shirama, N. Kato, and M. Kashino, “The Singular Nature of Auditory and Visual Scene Analysis in Autism,” Philos. Trans. R. Soc. Lond. B Biol. Sci., Vol. 372, No. 1714, 2017. pii: 20160115. doi: 10.1098/rstb.2016.0115.