Virtual Reality-based Sports Training System and Its Application to Baseball

2.1 Features of baseball batting

A baseball game always starts when the pitcher throws the ball to the first batter. During a game, starting pitchers usually throw about 100 pitches, and batters typically have about three opportunities at bat. Therefore, the batters have to adjust to the way the pitcher throws the ball during the game. Radio broadcasters commentating on the game are likely to report on how batters adjust to the way the pitcher throws the ball the second or third time they face the pitcher. Batters can therefore use what they have learned from virtual reality (VR)-based technology to improve their batting performance.

2.2 Definition of requirements

We held discussions with professional players and coaches to determine the necessary capabilities of the system. We extracted the following four requirements:

(1) The system can provide the ball trajectory with depth information.

(2) The system can reproduce the correct ball trajectory at arbitrary positions within the batter’s box.

(3) The system can be taken to any place and set up easily.

(4) The system can be updated with the latest information about the pitcher.

In the next section, we describe the system we propose that fulfills these requirements.

4.1 Billboard-based pitcher representation

There are several possible ways to represent a pitcher in 3D virtual space. In one 3D CG example, the pitcher’s posture and texture are measured. However, it is difficult to stably update VR content using this solution because pitchers will almost never permit their posture to be captured with a motion capture system. This is to prevent the opposing team from getting hold of such content and watching them before the game.

A well-known method called billboard-based video representation [1] does not rely on the pitcher’s posture. Roughly speaking, one advantage of this method is that it mainly makes use of captured videos. Another advantage is that it can easily represent subtle nuances in pitchers such as changes in their facial expressions, which are very hard to measure and represent through the use of CG. We used these advantages to provide billboard-based representations of pitchers. First, we captured a video of the pitcher’s motions from outside the playing field. We then placed a virtual panel (called a billboard) at the pitcher’s location in a virtual stadium depicted by 3D CG. Finally, we played the captured video on the billboard. In baseball, the positions of the pitcher and batter do not significantly change. Thus, the billboard-based representation suits this task.

4.2 Representation of ball trajectory by CG

Unlike a pitcher’s motion, the thrown ball is a significantly difficult target for the billboard approach to represent. This is because it includes large and abrupt position changes; a thrown ball moves about 18 m in about 0.5 s. Therefore, we do not use captured video directly to represent the ball. Instead, we first measure the ball trajectory and then use CG to render the ball. This solution enables us to provide an accurate ball trajectory at arbitrary positions from the batter’s box area.^*

However, a severe disadvantage occurs in combining billboard-based pitcher representation and CG-based ball rendering. That is, there is a duplication between the actually filmed ball and the CG rendered ball (Fig. 3(a)). To avoid this duplication, the filmed ball should be removed by image processing. We used the inpainting method proposed by Isogawa et al. [2] to remove the ball in the filmed video. We verified that it significantly improves the quality of experience, as shown in Fig. 3(b).

Fig. 3. Duplication of the filmed ball and the CG rendered ball, and the same image after removing the filmed ball.

5.1 Acceptability

There are various indexes to evaluate batters; one of the most popular is batting average. However, we obtained users’ (i.e., batters) subjective evaluations in a system trial conducted throughout a baseball season. This made it hard to make comparisons under a controlled setting. Furthermore, the task of successfully hitting a baseball is a quite uncertain one; a batter who gets a hit only three times in ten attempts is considered to be a very good hitter. This makes it very hard to evaluate the performance of our system on the basis of an objective score such as batting average.

(1) Depth sensing

Almost all professional baseball players have used videos to check out their opponents. However, watching videos on a 2D display does not give sufficient depth information since it provides neither binocular parallax nor motion parallax. In contrast, the system we propose reproduces 3D ball positions in virtual space and renders videos depending on the position and orientation of the HMD. It also renders them for both the right eye and left eye positions. Thus, it provides an HMD experience with depth cues, that is, binocular parallax and motion parallax. Many athletes who used our system said that with it, they really felt that the ball was coming from the pitcher.

(2) Simulation training for adjusting to pitcher’s motion

Many batters say that it is very helpful for them to be able to see simulations of the opposing pitcher before the game. This enables them, for example, to check on how they will adjust to the pitchers’ motions based on the changes the pitchers make in the speeds at which they throw the ball. They feel that this will allow them to make full use of their batting chances, even in their first time at bat.

(3) The system as a communication device

We received a comment from a staff member who said that using this system made it possible to expect better communication with players since both coaches and players could get the same ball trajectory.

However, some points arose that need to be addressed, including the low resolution provided due to the use of the HMD and the insufficient field of view for batting. One of the tasks we will tackle in the future is to carry out a more detailed analysis of the system. The most important but most difficult thing we need to achieve is how to objectively measure the perceived correctness of the batter’s experience. We will need to tackle this point to further develop the system.

5.2 Operation ability throughout a season

We conducted a trial of the system throughout the 2016 baseball season. The season comprised 143 games, but some of them were lacking in data to create VR content. Nevertheless, we created VR content for 96 pitchers, who threw an average of 15.6 pitches per game.

A histogram of the elapsed time between a request from a team for content and the time we returned the completed VR content to the team is shown in Fig. 4. Note that we could have returned the content sooner than we did if it had been necessary to meet the team’s schedule. Sometimes, due to schedule changes or other factors, we were requested to provide content within a short time, even within a single day. The relatively high per-day submission frequency shown in the figure indicates that with our system we were able to handle such requests appropriately.

Fig. 4. Elapsed time between receiving a request for content from a team and the provision of the content.