Elucidating the Brain Processing Mechanisms of Athletes Using Virtual Reality Technology

1. Introduction

An athlete’s physical fitness, skill, and state of mind are essential factors in sporting performance (Fig. 1). Fitness, including muscle strength and cardiovascular capacity, has long been studied in the fields of exercise physiology and sports medicine, and these findings have been utilized in various ways in sports coaching and training. The athlete’s skill and mental state are also critical in terms of success in actual sports games. Examples of these include dexterously coordinating the various parts of the body, adequately recognizing a given situation, and overcoming mental pressure. These abilities are mainly determined by information processing in the brain. However, the mechanisms involved are poorly understood because work in this area faces a number of technical limitations. Thus, systematic training methods designed to enhance an athlete’s skill and state of mind have not been fully established.

Fig. 1. Scope of sports brain science project.

NTT is engaged in a sports brain science project to improve athletic performance (Fig. 1), where the aims are to read the key brain processing features related to an athlete’s skill and mental state, develop assistive methods and devices based on the obtained findings, and finally, to improve the athlete’s brain functions. Information and communication technology (ICT) know-how as well as neuroscientific techniques are required for the success of this project. For example, wearable sensing and information engineering technologies will be important if we are to effortlessly obtain behavioral and biological data and effectively feed the obtained findings back to the athlete. The virtual reality (VR) technology introduced in this article will also be a powerful tool for facilitating our project, and especially in elucidating the brain processing mechanisms related to an athlete’s skill and mental state.

2. Utilizing VR for sports brain science

To identify the key features of brain processing related to skill and mental state, it is first necessary to measure the behaviors and biological signals (e.g., movements, muscle activity, heart rate, and respiration) of an athlete engaging in a sport. There are two possible ways to do this. One is to perform measurements in a laboratory, which is the most commonly used approach. Such measurements enable the conditions to be strictly controlled, but there is a potential limitation that the target action will be unlike the actual sports action.

The second way is to perform measurements during an actual sporting event. Recent rapid advances in various types of wireless and wearable sensors have gradually enabled us to obtain behavioral data in actual sports fields [1]. We also attempt to obtain biological data during sporting activity using wearable bioelectrodes such as the conductive fabric called “hitoe” [2]. However, unlike in the laboratory, it is hard to control the measurement conditions and situations during an actual sporting event. Since there are usually many factors in actual sports environments that can potentially influence an athlete’s performance, it may be difficult to interpret the observed phenomena. Consequently, there is a trade-off between controllability and reality with both types of measurement. Therefore, the measurement approach should be selected according to the aim of the study.

Here, we propose the use of VR technology as a third approach to sports measurement that can compensate for the trade-off between controllability and reality. VR has the advantage of providing a highly realistic visual experience. Various experience-based services and exhibits have recently been introduced and are often combined with the use of a highly immersive head-mounted display (HMD) to enable the user to experience situations such as standing in a high place or riding a roller coaster, and they have successfully monitored user responses. There have also been some sports-related applications, including an HMD-based VR system that enables the user to experience a live 360-degree view of a football player (STRIVR Labs, Inc.). Another advantage is that the technique allows us to arbitrarily manipulate a visual environment combined with computer graphics (CG), which makes it possible to configure a visual environment that is inconceivable in the real world (Table 1). With these advantages of VR technology, we are trying to construct VR environments for achieving sports measurement capabilities with both sufficient reality and flexible controllability of the visual experience so as to capture behavioral data that were formerly hard to observe.

Table 1. Advantages and disadvantages for sports measurement environments.

3. Measuring behavioral data with sports VR system

A baseball batter has to manipulate his/her bat within very tight time constraints (within about 0.5 s) from the release of the ball by the pitcher to the moment of impact [3]. However, it would appear to be impossible to achieve appropriate batting by employing the relatively slow conscious (explicit) processing in the brain, because it takes 0.5 s or more between recognizing the ball trajectory to swinging the bat. A key requirement is fast unconscious (implicit) processing in the brain without conscious perception or judgment. Instant decision-making and the quick execution of an action are critical in many sports in addition to batting in baseball. We are studying such implicit brain processing mechanisms with VR technology using baseball batting as an example.

NTT Communication Science Laboratories and NTT Media Intelligence Laboratories have developed an HMD-based VR system for baseball batting (Fig. 2), in which a user can experience virtual batting from a batter’s viewpoint in a virtual baseball stadium constructed from a combination of real images and CG [4]. The pitched balls are depicted by CG based on previously recorded ball trajectories and are thrown in time with the motion of a pitcher in a simultaneously recorded video. The batter and the bat movements are measured using the nine axes of inertial sensors attached to various parts of the body and the bat and are represented in a virtual avatar in real time. Visual images presented to the right and left eyes are independently displayed on the HMD in tune with the position and direction of the user’s head, which allows the user to obtain a virtual view with appropriate depth sensation in an arbitrary location and orientation in the batter’s box. These VR system settings enable a user to experience a strong feeling of reality, as if standing in the batter’s box, and to perform virtual batting in which the ball bounces depending on its interaction with the bat.

Fig. 2. VR system for experiencing baseball batting.

Examples of behavioral responses to virtual batting by a single expert and a beginner are shown in Fig. 3. The participants were instructed to swing at fastballs or curve balls that were thrown in the strike zone. The head and foot movements and the respiratory waveforms for each of 10 trials when randomly presenting fastballs that were strikes are shown in Fig. 3(a). The head movement of the expert participant changed smoothly before the pitcher released the ball, while this was not the case with the beginner. This suggests that the expert prepares in anticipation of the ball release, while the beginner reacts to the ball release. The expert also synchronized his respiration and his foot stance to his estimated timing of the ball’s impact. Further, we found that when a ball was coming toward the head, which is known as a beanball (red lines in Fig. 3(b)), the expert moved his head backward and avoided the ball, but the beginner moved too late or insufficiently to avoid the ball.

Fig. 3. Behavioral responses to virtual baseball batting.

These behavioral data not only reveal certain characteristics of an athlete but also provide unique findings via VR measurement. For example, it is not ethically possible for researchers to have a pitcher intentionally pitch a beanball at a batter in a real game, and hence, to the best of our knowledge, there is no way of finding out about such occurrences. We also want to obtain novel findings about how a batter predicts a ball and how emotional experiences (e.g., fear) affect batting history by occasionally removing the ball and manipulating the sequence and variety of pitches.

4. Future prospects

Work has begun recently on evaluating sports performance in virtual environments [5]. However, most of these studies have aimed at approximating the real world, while there have been few studies that purposely manipulated visual environments. Also, the observed objects are mainly behavioral outcomes (e.g., movement and force) with few underlying biological data. Although the VR system we describe in this article is confined solely to limb movements and respiration, its combination with other measurements such as muscle activity, eye movements, and heart rate should enable us to understand multiple aspects of sports performance such as how force is exerted, lines of sight, and the athlete’s mental state. We anticipate that our work using VR technology will clarify the key physical and mental features of an athlete and the underlying information processing by the brain, which have not been determined with existing approaches in the laboratory or in real sports fields.

Our VR system will be useful not only as a tool in sports neuroscience but also in sports training and coaching. For example, although a batter can carefully study opposing pitchers using video, realistic experiences of pitches from the batter’s viewpoint would be more useful. Furthermore, sharing these VR images with coaches can result in more practical and effective coaching. A batter can also use batting training to deal with weak pitchers and pitches. Moreover, this interactive VR technology will be able to provide new value and content in relation to other fields such as rehabilitation and entertainment.

Acknowledgment

This research is supported by the Japan Science and Technology Agency (JST) CREST project.

References

	Toshitaka Kimura Senior Research Scientist, Human Information Science Laboratory, NTT Communication Science Laboratories. He received a Ph.D. in human neuroscience from the Graduate School of Arts and Sciences, the University of Tokyo, in 2003. He joined NTT Communication Science Laboratories in 2003. He has been engaged in research on sensorimotor control and action-perception interaction in humans. He has recently been focusing on brain functions in athletes and the use of ICT in sports. He is a member of the Society for Neuroscience, the Japan Neuroscience Society, and the Institute of Electronics, Information and Communication Engineers (IEICE).
	Kosuke Takahashi Research Scientist, Visual Media Project, NTT Media Intelligence Laboratories. He received a B.Sc. in engineering and an M.Sc. in informatics from Kyoto University in 2010 and 2012. He joined NTT Media Intelligence Labs in 2012. His research interests include computer vision. He received the Best Open Source Code award Second Prize at the 2012 IEEE Conference on Computer Vision and Pattern Recognition.
	Dan Mikami Senior Research Scientist, Visual Media Project, NTT Media Intelligence Laboratories. He received a B.E. and M.E. from Keio University, Kanagawa, in 2000 and 2002, and a Ph.D. from Tsukuba University in 2012. He joined NTT in 2002. His current research activities are mainly focused on computer vision and virtual reality. He was awarded the Meeting on Image Recognition and Understanding 2009 Excellent Paper Award 2009, the IEICE Best Paper Award 2010, the IEICE KIYASU Zen’iti Award 2010, the IPSJ SIG-CDS Excellent Paper Award 2013, and the IEICE Human Communication Award 2015.
	Makio Kashino Senior Distinguished Scientist/Executive Manager of Human Information Science Laboratory, NTT Communication Science Laboratories. He received his B.A., M.A., and Ph.D. in psychophysics from the University of Tokyo in 1987, 1989, and 2000. He joined NTT in 1989. From 1992 to 1993, he was a visiting scientist at the University of Wisconsin. He is currently a visiting professor in the Department of Information Processing, Tokyo Institute of Technology (2006–). He has been conducting psychophysical experiments and using neuroimaging, physiological recordings, and computational modeling to investigate the functional and neural mechanisms of human cognition, especially auditory perception, crossmodal and sensorimotor interaction, and interpersonal communication. His other recent research interests include sports, music, and autism.