Goalball × Ultra-realistic Communication Technology Kirari!

3.1 Highly realistic sound-image localization technology: Kirari!

Kirari! is a communication technology that enables spectators to experience watching a game as if they were in the competition venue, even if they are far from the venue [2]. With “goalball experienced through sound,” we created an acoustic experience by using highly realistic sound-image localization technology, one of the technical elements of Kirari!. This technology reproduces the spatial and physical wavefronts of sound on the basis of a physical model [3]. It separates the sound collected with a single microphone at the tournament venue and reproduces it as if a specific sound were generated at an arbitrary position. A key point of this technology is a group of speakers (speaker array) closely arranged in a straight line. By adjusting the playback timing and power of the sound radiated from each speaker to focus the sound at an arbitrary position, a sound field is reproduced as if a sound source existed there.

The configuration of “goalball experienced through sound” is shown in Fig. 3. Due to operational constraints, we could not set up our own cameras to transmit the event, so we decided to use broadcast video. Video images of the game shot at the venue of a goalball tournament are transmitted to a remote site as broadcast waves via the broadcasting equipment of the broadcasting station. At the remote site, the position of the ball and players as well as the information about the type of ball thrown (acquired from the received broadcast-wave images) are input in real time, and an acoustic space is created on the basis of that spatio-temporal information (Fig. 4).

Fig. 3. Configuration of the entire system.

Fig. 4. Real-time acoustic-space reproduction.

We aimed to extract the locations of the ball and players, as well as information about the type of ball thrown, from the broadcast video. However, the cameras and angle of view change frequently during broadcast video, so we adopted a user interface (UI) through which the ball position is input manually (Fig. 5). Two people, one in charge of team A and one in charge of B, each used a tablet with the UI installed to watch the broadcast video and enter the starting position and time of each team’s attack, type of throw (bouncing ball or ball along the ground), and result of the throw (blocked, goal, or out of bounds) with the push of a button. From the input result, sound information is generated from a sound-source file of the throwing sound prepared in advance. That information is then processed using Kirari! via the network to reproduce the ball-throwing situation in real time.

Fig. 5. Real-time input UI for ball-throwing information.

3.2 Acoustic production with inclusive design

(1) Selecting sounds to localize as the sound image

To reproduce the acoustic space that the players hear on the court and allow non-players to fully experience the players’ excellent sense of hearing, we used an inclusive design method, which involved goalball players and the visually impaired in the upstream design process. Specifically, we defined and created the requirements for the sound through a process of interviews, observation of ball-throwing situations, and acoustic evaluation (Fig. 6). This process enabled us to understand the various sounds heard on the goalball court, but distracting sounds made by the opposing team (such as the sounds made to obscure the throwing position by hitting the floor) were included but are ignored by the players. In other words, the players always search for the ball by following a series of throwing sounds such as the run-up sound of the player holding the ball and that of the ball hitting the floor in the manner of a single stroke. Accordingly, we decided to focus on the throwing sounds (run-up sound, ball sound, and result sound of ball throwing such as blocked shot or goal) and reproduce those sounds. That is to say, instead of trying to reproduce the acoustic space of the physical tournament venue as is, we attempted to reproduce the sounds that the players selectively perceive from the various sounds that they hear. By doing so, we aimed to allow non-players to experience the simulated exceptional auditory sense of the players.

Fig. 6. Players experiencing “goalball experienced through sound”.

(2) Simplification of sound-image localization

For the path of the thrown ball, the players distinguish the distance from one end of the goal to the other (9 m) in nine segments (each with width of 1 m). However, the spatial resolution of sound of non-players is not as good as that of players, so it is difficult for non-players to determine the route of the thrown ball when the sound image is localized as is. By reducing the spatial resolution in the goal direction from nine segments to three (left, center, and right), we improved the non-players’ recognition rate of the ball position by sound.

We also created an introductory content to give a lecture on sound localization to the non-playing spectators so they could get used to sound localization. In preparation for experiencing a game, the sound of throwing the ball along various paths was localized. Non-players commented that they could gradually determine the path of the thrown ball just by listening to the sound. This suggests that listening the introductory content before the game was useful to understand the game. Therefore, by simplifying the path of the thrown ball and creating the introductory content, we were able to create a system that allows non-players to experience the sounds that the players hear during a game.