Muscle Interface Technology for Augmenting Human Capabilities through Efficient Motor Skill Learning by Electrical Muscle Stimulation

Muscle interface technology for augmenting human capabilities through efficient motor skill learning

—Dr. Niijima, what exactly is “muscle interface technology for motor skill learning?”

In a few words, “muscle interface technology for motor skill learning” is a system that supports motor skill learning via muscles in sports, playing a musical instrument, and other endeavors. In the research that I’m now engaged in, I’m targeting piano playing. On analyzing the movements made by beginners and experts with sensors, the data obtained shows that the physical movements they make when playing the piano are completely different from each other. One reason why beginners cannot play the piano like experts is that they apply more force than necessary or fail to use their muscles in the right way. With this being the case, I thought about developing a system that, based on these physical differences between beginners and experts, could enable beginners to directly experience the movements of experts by applying electrical stimulation to their muscles.

For example, conventional motor skill learning has been centered on the visual and auditory senses in which an expert may verbally teach beginners how to move their bodies correctly or beginners may simply observe an expert’s performance. However, learning an expert’s muscle movements by only visual or auditory means can be difficult. The aim of muscle interface technology is to enable efficient learning of correct muscle movement by moving a user’s muscles in a more direct manner.

While at university, I played Sports Chanbara and came to teach newly enrolled college students and even elementary school students. It was at that time that I learned that what I first had to teach all beginners was “relax when you swing your sword.” Additionally, when I was learning how to swim, the instructor said the same thing: “Please relax.” Moreover, when studying how to play the piano from a book, the instruction “Try to relax when playing!” was likewise included. I thought it was interesting that “please relax” was a key point in learning just about any movement. So having some interest in motor skill learning, I began my research thinking, “I wonder if motor skill learning could be supported by computer.”

—What kind of method do you use to implement muscle interface technology?

In this muscle interface, we use a technology called electrical muscle stimulation (EMS) to move muscles by electrical stimulation to teach beginners how experts move their muscles. In this regard, a muscle normally contracts by receiving an electrical signal from the brain, but EMS applies electrical stimulation from an external source to the muscle instead of an electrical signal from the brain.

This technology has been traditionally used in the medical field for rehabilitation purposes, but I began my research thinking that EMS has probably never been used as an interface for healthy individuals to learn motor skills. Let me describe how motor skill learning in piano playing can be supported by EMS using Fig. 1. In piano playing, there is a technique called “tremolo” in which the player rapidly alternates between two or more keys. For a beginner, this technique is seemingly easy but actually difficult, and as a result, a beginner get tired easily, preventing him or her from playing the piano over an extended period of time. To investigate the reason for this, we used electromyography sensors to analyze how beginners and experts use their muscles while playing the piano. We found that beginners apply more force than necessary to the extensor digitorum and extensor pollicis longus muscles when moving their fingers compared with experts. In contrast, experts use their supinator and pronator teres muscles for wrist rotation instead of using muscles to move their fingers. As a result, they do not get tired easily and can play the piano for a longer period of time. To change the way that beginners use their arm muscles, we created a system that teaches beginners how to play a tremolo while rotating their wrist like experts by applying EMS to the supinator and pronator teres muscles alternately. By practicing with this system, a beginner becomes increasingly aware of wrist rotation instead of finger movements, and with a little practice, learns the knack of playing tremolos for an extended period of time without fatigue.

Fig. 1. Example of muscle interface technology for motor skill learning in piano playing.

The key point here is that the beginner moves his or her hand in time with the motion of the electrical stimulation thereby learning in an active manner. In this way, motor skill learning progresses, and after practicing, the beginner can play tremolos without EMS using wrist rotation in the right way.

In conventional EMS technology used in human interfaces, there have been many studies on contracting and moving muscles by EMS but there has been practically no research on reducing force applied to a muscle. In contrast, our muscle interface technology indirectly transcribes a relaxed state of specific muscles by applying electrical stimulation to a beginner’s muscles based on data obtained on efficient body movement by experts.

The most difficult thing about this research is determining the extent to which complicated movements can be taught. For example, a full-body motion like a golf swing involves multiple muscles, so helping a beginner to learn a golf swing becomes a complicated problem. In such a case, the crux of this research is determining whether expert movements can be taught by applying electrical stimulation to certain muscles to produce certain movements. To this end, we must first obtain knowledge from research fields dealing with physical movement and then collect an extensive amount of information from all sorts of media in addition to the literature. We can then proceed to extract the ways that certain movements should be made common to these sources of information. After this, we can turn biological signals during movements into data using electromyography sensors or motion capture and analyze which muscles are used in what ways. The method of analysis here is not only to look at individual muscle data. A technique called “muscle synergy analysis” can also be used to analyze muscle coordination, that is, to analyze which muscles cooperate with which other muscles. By collecting diverse types of data in this way, we can determine which muscles we should focus on to reproduce expert movements. In addition, we don’t stop at this step of collecting data. In the end, it is crucial that we create a system for actually applying electrical stimulation and try it ourselves, and that we conduct subjective evaluations on whether this system will enable users to experience the feeling of motor skill learning.

Future of muscle interface technology in breaking down current barriers

—What is your outlook for muscle interface technology in the years to come?

Figure 2 shows how cybernetics technology such as muscle interface technology that I am researching can be linked to a human augmentation platform for controlling human movement via the network. As a future outlook, I believe that it will be possible to support motor skill learning remotely in real time by using movement transcription technology that copies human movement to another person. This technology transmits data on an expert’s body to a beginner at a remote location so that the beginner can receive stimulation by electrical stimulation equipment.

Fig. 2. Outlook for muscle interface technology linked with a human augmentation platform.

In addition, I think it would be great if muscle interface technology could be used to extend healthy life expectancy. That is, when a person’s muscular strength and cognitive abilities begin to decline with age and what he or she could do when younger can no longer be done, muscle interface technology could be used to synchronize past and present movements by downloading old data and restoring the way that one used to move. When a person stops trying out new things, that person’s lust for life will quickly be lost. Supporting one’s learning through a muscle interface can overcome the first barrier to skill acquisition and enable people of any age to take up the challenge of doing new things. In this way, I would like to help people with their lives.

—Dr. Niijima, please leave us with a message for other researchers, students, and business partners.

I currently belong to NTT Human Informatics Laboratories, which, as the name implies, is focused on research centered about humans. Engineers and researchers at times tend to focus their efforts on improving technology while leaving people behind. However, if one stops for a moment and considers the actual purpose of research, it is certainly to “improve people’s lives and future.” As a researcher of human-computer interaction, I myself treat humans as the target of my research, so I can identify with the vision of NTT Human Informatics Laboratories, which I feel to be a very pleasant working environment.

A key strength of NTT is the ability to talk freely with experts in diverse fields within the company. It is said that my research field, human-computer interaction, is a scientific field that is not defined by specialized technology. When I began my research, the knowledge and technology that I needed spanned a wide range. For example, in motor skill learning that I am now researching, not only do I need sensor technology, analysis technology, and engineering skills for combining a series of systems, I also need knowledge related to human movement from the field of neuroscience as well as information on various types of muscles. In short, it is only by gathering such knowledge and technologies from disparate fields that I can research a single theme. At this time, surveying all of this information by myself would take a considerable amount of time and labor, but at NTT, it is easy to ask other researchers for assistance or to hold meetings with them, so specialized knowledge can be acquired in a relatively short time. I also feel that the value of the NTT brand name and the size of the company are amazing. It is often the case that researchers fulfill their role by conducting research, writing papers, and presenting them somewhere at academic societies. At NTT, however, announcing new technologies developed for the good of society draws a response several orders of magnitude greater than usual, which can lead to the actual implementation of those technologies in society. In my opinion, only NTT can provide such a sense of accomplishment and sense of worth.

I’ve recently noticed that there are young researchers and students that are conducting research while still not knowing what kind of research they personally like. What I have learned over the years in my research is that it’s important in the end to do what you enjoy. To give an example, my research during my university and graduate school days was like a short-distance sprint—even if I wasn’t particularly interested in that research, it consisted of just short-term objectives, so I gave it my best effort. On the other hand, once I entered adult life as an actual employee, I could imagine my research continuing on for several decades rather like a long-distance run, and I thought that it would be difficult to stick with any research that I had no real interest in. So on thinking on how I could run for such a long distance, I realized that doing something because you truly like doing it could be a powerful driving force. Everyday, there are a variety of things that catch my interest, and it often happens that I begin research on some topic thinking, “This is really interesting!” Of course, a desire to solve social problems is important, but I also think it’s important that the individual take a good look at what he or she inherently likes and to verbalize that preference clearly.