Understanding and Shaping the Athlete’s Brain
—NTT Sports Brain Science Project—

1. Importance of the brain in sports

What are the necessary conditions for becoming an outstanding athlete? It goes without saying that physical abilities are important such as a resilient body excelling in muscular strength and cardiopulmonary function and an appropriate form that can effectively produce power and minimize the risk of injury. However, to reach even higher levels as an athlete, physical abilities by themselves are not sufficient. There is also a need for other faculties such as the ability to grasp match conditions, develop a strategy and make instantaneous decisions, the ability to predict your opponent’s next move based on current behavior and take preemptive action, the ability to deceive and manipulate your opponent, the ability to quickly and flexibly adjust one’s actions even during complex moves, and the ability to amply demonstrate one’s strengths while under intense pressure during a big match. None of these abilities can be achieved without advanced information processing in the brain. In general, top athletes are especially proficient in such brain functions.

Research in conventional sports science and associated training techniques has led to an accumulation of substantial results from a physical point of view. However, brain functions in relation to sports are still largely unexplored even from a global perspective. Against this background, NTT launched its Sports Brain Science project (SBS) in January 2017 as an interdisciplinary research organization [1]. The purpose of SBS is to understand the brain functions that support superior performance in top athletes and to actually improve the performance of athletes based on those scientific findings. Now, with about one year behind us, we are off to a good start, having gained the cooperation of top-level athletes and teams centered on baseball and softball. Here, I would like to introduce the basic concepts and research policy of SBS.

2. Deciphering implicit brain functions

In sports where athletes play against an opponent, for example, baseball and martial arts, the time allowed for grasping ever-changing conditions, making decisions and predictions, and adjusting movements is exceedingly short, on the order of split seconds. For the athlete, this is too short a time to recognize or consciously control something that occurs in the brain. Out of necessity, important brain functions in sports with an opponent are for the most part unconscious activities. From the athlete’s perspective, the feeling is that the body reacts on its own.

These brain functions, however, are different from spinal cord reflexes and other simple reactions common to everyone. Rather, they are extremely advanced functions the knowledge of which would not be possible without the results of a massive amount of learning about the behavior of opponents and physical objects such as balls and the way the body moves in response to that behavior. Such brain information processing affects complex actions while being unconscious to the athlete, and for this reason, we refer to different instances of such processing as implicit brain functions. We have been involved in the basic (i.e., not directly sports-related) research of implicit brain functions for many years [2]. The first objective of SBS is to clarify how the various elements of these implicit brain functions related to sports performance differ between superior athletes and regular athletes.

However, implicit brain functions by their very nature prevent an athlete from being aware of them, so asking a top athlete about them is no help in arriving at the truth. It is rare for a person excelling at a certain skill to be able to accurately put into words how he or she does it. Even if the skill can be verbally described, it is not uncommon for those words to describe not what actually occurs but rather to fit the results observed. This is thought to be one of the factors behind the difficulty of coaching a specific skill. A great player does not necessarily make a great coach.

With this being the case, SBS has been using body-mind reading technology developed at the NTT laboratories with the aim of deciphering the implicit brain functions of athletes (Fig. 1). The first step in this technology is to measure the movement of various parts of the athlete’s body in action and various biometric signals. Of great importance here is that such measurements be made under conditions as close as possible to those of a real match without impeding the athlete’s inherent performance. Here, we use multiple video cameras and wearable inertial sensors to record body movements and quantify them. We also use wearable sensors to measure biometric signals such as brain waves, cardiac potential and heart rate, respiration, and myoelectric potential. Additionally, we use camera goggles to track eyeball movements.

Fig. 1. Deciphering and adjusting implicit brain functions.

The second step is to use the data obtained from these measurements to extract essential information on the physical behavior and brain information processing of athletes. Here, to obtain information on brain information processing, the brain activity does not necessarily have to be measured. For example, eyeball activity (direction of line of sight, fine eye movements called microsaccades, changes in pupil diameter, etc.) reflects and changes the internal brain conditions (surprise at something unexpected, direction and extent of attention, information processing load, etc.). That is to say, if eyeball activity can be analyzed, it should be possible to infer to some extent internal brain conditions at that time [3]. Similarly, changes in heart rate and respiration, for example, can provide information related to underlying brain activity more so than the movement of certain parts of the body.

To carry out such analysis, we are applying various artificial intelligence related technologies from NTT’s corevo^TM initiative including image processing, biometric signal processing, and machine learning. In all, these analysis technologies constitute body-mind reading, which is becoming a powerful tool in the hands of SBS.

Appropriate experimental environments are needed in order to conduct experiments that can capture the essence of sports. In SBS, we divide experimental environments into four main types (Fig. 2). The first is an actual match. The stress that accompanies the need to perform cannot be felt outside of an actual match. In the case of baseball games, we attach a wearable sensor (“hitoe”) under the athlete’s uniform to capture heart rate, acceleration, and other data, and we analyze how psychological factors can influence performance. However, the fact that an actual match is literally real also means that many factors including accidental occurrences and things that cannot be identified will be intertwined in a complex manner. As a result, the experimenter does not have control over everything in such an experiment.

Fig. 2. Four types of experimental environments.

With this in mind, the second type of experimental environment is the smart bullpen that we created to extract at least a portion of the true nature of sports while enabling a controlled experiment. Details are provided in another of the Feature Articles [4] in this issue, but in brief, the smart bullpen is an experimental facility equipped with multiple cameras and various measurement devices in a space such as an indoor baseball practice area. A key feature of this facility is the ability to analyze in detail the mutual interaction among multiple athletes, as in the battle between a pitcher and batter in baseball and softball. The essence of competitive sports lies in the mutual interaction between players. Simply analyzing pitching or batting will not lead to an understanding of what it takes to win. The smart bullpen enables the physical movements and biometric signals of both the pitcher and batter to be measured synchronously under competitive conditions while also enabling the behavior of the ball (velocity, spin, trajectory, etc.) to be simultaneously recorded by specialized radar. An example of an experiment using this smart bullpen is described in another Feature Article [5] in this issue.

The third type of experimental environment is virtual reality (VR). NTT has developed a VR system that enables the user to experience the pitcher’s form and the ball’s trajectory from the batter’s line of sight in three dimensions [6, 7]. With this system, we are able to create experimental conditions that cannot technically or ethically be produced in a real environment such as the smart bullpen. For example, a VR system enables us to manipulate information on the pitcher’s form or ball movement for a certain purpose or to create a dangerous situation such as a ball flying toward the batter’s head and to then analyze the batter’s response to such situations.

The fourth type of experimental environment is a laboratory for conventional cognitive brain science. In this environment, we can present precisely designed visual, auditory, and haptic stimuli and accurately measure arm and eyeball reactions, brain activity, and other data. Here, a participant is fixed in a chair facing equipment and asked to deal with very simple tasks, so this environment is far removed from an actual sports scene. On the other hand, it is geared to fundamental experiments that enable specific factors to be isolated and analyzed in detail.

As described above, these four types of experimental environments each have their advantages and disadvantages. Which to use should be decided based on the current research objective.

3. Adjusting implicit brain functions

The second objective of SBS as described above is to actually improve the performance of athletes based on scientific knowledge related to the implicit brain functions of top athletes. To this end, there is a need for some means of adjusting implicit brain functions in order to improve performance. Conscious control of implicit brain functions is difficult, and verbalizing them is likewise hard. As a consequence, giving verbal instructions to athletes can hardly achieve results. The athlete needs to objectively grasp his or her state of performance and intuitively understand points to be corrected or the direction of change. For this reason, we are working at SBS to present information on movements through sensory feedback by visual or auditory means.

For example, we can use a large-screen display installed in the smart bullpen to present video of a pitcher’s form captured with multi-angle cameras together with data such as the ball’s speed and spin several seconds after the delivery. In this way, pitchers can check their movement and the result directly after throwing the ball while their sensation of the pitch still remains. Users have commented that this system helps them to become aware of the gap between subjectivity and objectivity, which is a common problem for athletes, while providing them with an opportunity to make appropriate corrections.

We have also developed a sonification system that converts actions such as pitching a ball into sound based on information received from wearable myoelectric sensors and acceleration sensors attached to various parts of the body (Fig. 3) [8]. While differences in form can be easily understood by video, the amount of applied force and the timing can be easily understood by sound. This system can therefore help the athlete to intuitively understand the difference between a model action and one’s own actions, and the difference between one’s good form and poor form. We will continue to research methods of intervention using VR.

Fig. 3. Sonification of pitching action.

4. Future outlook

In SBS, our two objectives of understanding an athlete’s implicit brain functions and improving performance constitute our basic policy around which our work revolves. It is a general rule that a big gap will exist between basic research in the laboratory and improving athletic performance in the field. To overcome this gap, we have adopted a style that enlists the cooperation of top athletes and teams to develop an awareness of real problems in the field and uncover seeds for basic research. In this way, we can promote athlete-centered research and not just research for research’s sake. We must strictly avoid having a negative effect on athletes by simply intervening. To intervene on scientific grounds, a large volume of high-quality data is needed, and at the same time, a cooperative relationship with athletes is essential.

Progress in research in SBS will help to systemize a methodology for winning based on scientific grounds in relation to brain characteristics. As part of this progression, there will no doubt be findings that diverge from what has so far been considered common sense. Inside a facility like a smart bullpen, athletes can develop an understanding of their own state, including things that they are not normally aware of, and identify points for improvement. Such a facility may also help to change one’s behavior or form or detect signs of injury. In addition, accumulating data on many athletes will make it possible to diagnose individual type, aptitude, level, and other factors. It will also facilitate coaching oriented to individual conditions and enable early discovery of talent.

In SBS, we are presently focusing on the sports of baseball and softball, but we feel that the knowledge gained here can be essentially expanded to other sports. Furthermore, we are now targeting only top-level athletes such as those in professional baseball and the Japanese national softball team or an equivalent level, but in the future, we would like to broaden our target level to include junior athletes and even devoted sports fans and the elderly. More details on our research and progress to date can be found on the SBS website [9].

Acknowledgment

Part of this research received CREST funding (JPMJCR14E4) from the Japan Science and Technology Agency (JST).

References

[1]	Advertisement feature: “Split Seconds Matter – the Brain and Sport,” Nature, Vol. 549, No. 7670, 2017.
[2]	M. Kashino, M. Yoneya, H.-I. Liao, and S. Furukawa, “Reading the Implicit Mind from the Body,” NTT Technical Review, Vol. 12, No. 11, 2014. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201411fa6.html
[3]	S. Furukawa, M. Yoneya, H.-I. Liao, and M. Kashino, “The Eyes as an Indicator of the Mind—A Key Element of Heart-Touching-AI,” NTT Technical Review, Vol. 14, No. 5, 2016. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201605fa4.html
[4]	M. Yamaguchi, “Sports Brain Laboratory for Measuring Athletic Competence and Performance,” NTT Technical Review, Vol. 16, No. 3, 2018. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201803fa2.html
[5]	D. Nasu, “Timing Adjustment of Baseball Batters Determined from Motion Analysis of Batting,” NTT Technical Review, Vol. 16, No. 3, 2018. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201803fa3.html
[6]	T. Kimura, K. Takahashi, D. Mikami, and M. Kashino, “Elucidating the Brain Processing Mechanisms of Athletes Using Virtual Reality Technology,” NTT Technical Review, Vol. 14, No. 11, 2016. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201611fa6.html
[7]	D. Mikami, K. Takahashi, N. Saijo, M. Isogawa, T. Kimura, and H. Kimata, “Virtual Reality-based Sports Training System and Its Application to Baseball,” NTT Technical Review, Vol. 16, No. 3, 2018. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201803fa4.html
[8]	T. Kimura, T. Mochida, T. Ijiri, and M. Kashino, “Body-mind Sonification to Improve Players Actions in Sports,” NTT Technical Review, Vol. 14, No. 1, 2016. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201601ra1.html
[9]	Website of Sports Brain Science project, http://sports-brain.ilab.ntt.co.jp

Trademark notes

All brand names, product names, and company names that appear in this article are trademarks or registered trademarks of their respective owners.