Body-mind Sonification to Improve Players Actions in Sports

3.1 Sonifying segmental coordination

For most sports, it is critical to coordinate multiple body segments sequentially. For instance, when pitching a baseball, a pitcher needs to operate his/her body segments in sequence from the leg to the arm, which is a well-known segmental kinetic chain [5], and a novice pitcher cannot perform this sequence well. Thus, we attempted to sonify such sequential patterns during pitching using acceleration signals recorded on the pitcher’s trunk, and then to apply it to pitching practice for novice participants.

An example of the sonification of the pitching sequence of the trunk in a single novice participant is shown in Fig. 1. We attached wireless acceleration sensors (Trigno, Delsys) at three different positions (P1, P2, and P3) on the back of the trunk (Fig. 1(b)) and detected three axes of acceleration signals at a 300-Hz sampling rate. We used analog acceleration signals during pitching to calculate two acceleration components in real time, namely, trunk rotation (A1) and tilting (A2); we calculated A1 and A2 from the P2 minus P1 signals and P3 minus P1 signals, respectively (top two panels in Fig. 1(c)). Then a sound (S) is synthesized as

S(t) = A1(t) S1 + A2(t) S2,　　　　　　　　　　(1)

where S1 and S2 indicate band-limited pink noise with peak frequencies of 100 and 500 Hz, respectively. The lowest panel in Fig. 1(c) shows a spectrogram of the generated synthesized sound. Although this spectrogram contains two unclear frequency components, if the player performed a good trunk motion he/she would be able to hear the two components separately (see Fig. 1(d)). Time 0 approximately indicates the ball release timing of pitching.

Fig. 1. Sonification of segmental coordination of the trunk during pitching.

The above sonification (auditory feedback) was applied while two novice participants engaged in pitching practice that consisted of repeating 100 non-dominant arm pitches with a fixed step length and moderate effort (Fig. 1(a)). In particular, they focused on trunk rotation and tilting, which are targeted in the feedback information, because novices frequently exhibit poor trunk motion [6]. They also performed other practice trials using visual feedback (6-sec.-delayed video streaming) in a separate session for comparison with the effect of auditory feedback. The order of auditory and visual practice sessions was counterbalanced between the two participants. At the beginning of each practice session, either the sound (Fig. 1(d)) or a video of an expert’s pitch was provided as a practice target. Before and after each practice session, we measured as a test session the chest, shoulder, and wrist motions in 10 trials without any feedback using a motion capture system (Oqus 300, Qualisys).

The mean velocity waveforms of three segments (chest, shoulder, and wrist) in each test session are shown in Fig. 2(a) for each participant. Both auditory (AF) and visual feedback (VF) practices improved the body sequence; namely, the sequential pattern from the chest to the wrist for peak velocity was clearer than in pre-practice trials. The inter-trial deviation of velocities for each test session is shown in Fig. 2(b). Interestingly, the auditory feedback showed less velocity deviation across test trials than the visual feedback. This result suggests that our real-time sonification for segmental coordination enhances the inter-trial repeatability of movements, compared with standard visual feedback.

Fig. 2. Effect of sonification of segmental coordination during pitching.

3.2 Sonifying coordination of muscle activity

The above sonification method expresses the kinematics of movement based on acceleration signals of the body. We propose here another sonification method for body state that describes the dynamics of movement, that is, the muscle activity pattern. Even if a player is made aware of a difference in motion by the sonification of kinematics, it is probably difficult to understand how to exert a force that will improve the movement because it is a theoretically ill-posed problem. Thus, we attempted to sonify the sequential muscle activity patterns, which more directly reflect the exerted force, based on multiple electromyography (EMG) signals from the trunk and the upper and lower limb muscles.

An example of the sonification of the muscle activity pattern of pitching is shown in Fig. 3. The generated sound consisted of multiple sound sources whose fundamental frequencies were different for every muscle and whose amplitudes were varied in proportion to the EMG activity (Fig. 3(a)). An expert (former professional) pitcher almost always exhibited a clear kinetic chain of muscle activity from the leg to the arm muscle (upper-left panel in Fig. 3(b)), while an amateur pitcher did not (lower-left panel in Fig. 3(b)), indicating unnecessary muscle activation. Therefore, the expert’s sound is very rhythmical (upper-right panel in Fig. 3(b)), while the amateur’s is slurred (lower-right panel in Fig. 3(b)). This sonification can be described as an instrument type of sonification, and a player can hear the difference in the muscle activity pattern as a melody. Using this sonification, a player can move so as to make as purposeful a sound as possible.

Fig. 3. Sonification of muscle activity pattern during pitching.

3.3 Sonification of excessive muscle activation

Some sports players cannot perform well because they try too hard, such as when putting while playing golf. This corresponds to the unintentional excessive activation of the muscles probably caused by mental pressure. That is, such excessive muscle activation reflects the state of mind. This phenomenon is very common and very serious in an actual sports match, but it is often hard to notice it. Therefore, it would be useful to make a player aware of this potential for excessive muscle activation.

An example of the sonification of overflow of muscle activity in the initial stages (address and take back) of putting in golf is shown in Fig. 4, based on EMG and acceleration signals of the wrist extensor muscles. The sound signal (S) consisted of a periodic waveform whose amplitude and fundamental frequency were varied in proportion to the short-term power of the EMG activity (E) divided by the acceleration (A) as follows.

S(t) = E(t) / A(t).　　　　　　　　　　(2)

Fig. 4. Sonification of excessive muscle activation during golf putting.

The sound mainly reflects the EMG activity when the acceleration was kept at almost zero. Therefore, the golfer could hear annoying sounds during the preparation (address) period when the muscle was activated more than necessary while the wrist moved less (lower panels in Fig. 4). Conversely, the golfer could not hear these sounds when the muscle was relaxed during preparation (upper panels in Fig. 4). Little sound was produced when the wrist was moving (take-back period) because the muscle activity and motion were coupled. This is a warning type of sonification, unlike the above sonification, and the player aims to produce as little sound as possible.