Efforts Concerning Enhancement of Human Ability to Achieve the Remote World

1. Efforts to enhance human abilities via the Remote World

To achieve the Remote World, we believe that it is not just a matter of connecting real spaces remotely, i.e., in cyberspace, but a world in which the real and cyber aspects are fused to create a new user experience. Our research targets education, especially education and training involving physical exercise such as playing musical instruments and sports. We aim to create new user experiences in the Remote World by enhancing human abilities by transmitting and sharing skills and physical techniques—regardless of time and place—that have been directly transmitted and shared between people in the real world.

We introduce two technologies to combine the remote and real worlds. The first is motor-skill-transfer technology, which collects objectively observable information, such as surface electromyography and electroencephalography on the skills and physical techniques (tips, etc.) of professional athletes and craftsmen, and transmits and shares this information directly to people using electrical muscle stimulation (EMS). The second is embodied-knowledge technology, which captures the subjective sensations that occur within a person and transmits and shares these sensations so that others can experience similar sensations.

2. Initiatives concerning motor-skill-transfer technology

We are developing motor-skill-transfer technology to create new forms of education and training in the Remote World. We are trying to create a world in which people from all walks of life can teach each other, regardless of time or physical distance, and gain unprecedented teaching and learning experiences, especially in learning about movement, which is a difficult task, even face-to-face. The concept of this technology is to reproduce and transfer the movement of a skilled person, that is, to support movement by sensing the human body’s motor control and intervening in it [1] (Fig. 1).

Fig. 1. Conceptual diagram of remote training by using motor-skill-transfer technology.

The relationship between motor control and motor support in regard to the human body is explained as follows. People move their bodies when motor instructions from the brain are transmitted to the muscles and the muscles contract. The brain then perceives and recognizes the results of the movement as stimuli through sensory organs and repeats planning and giving new movement instructions. In contrast, the combination of sensing and feedback locations, as well as the specific sensing and feedback methods, changes the content of motion support. Motor-skill-transfer technology supports communication and control in the human body through analysis and electrical-stimulation feedback on the basis of the sensing of biological signals such as brain waves and muscle potentials. Regarding sensing and intervention regarding motor instructions from the brain to muscles and the resulting motor activity, we have begun investigating two topics: (i) measuring muscle activity (via electromyography), sensing of motor status, and motor support (intervention) using EMS based on that motor activity and (ii) sensing and intervention regarding perception in the brain in response to input from sensory organs. We have also started to focus on vision, somatosensation, and vestibular sensation, which are important in postural control; the basis of movement.

We now discuss new forms of education and training using motor-skill-transfer technology. Typically, in the education and training concerning exercise, the instructor or trainer and the trainee are present face-to-face in the same space, and the trainee is instructed through words and gestures. In contrast, the technology of transferring motor skills through biometric sensing and intervention has two key features: (i) enabling instruction regardless of time and physical distance and (ii) creating a more-effective space (Remote World) that goes beyond face-to-face education and training. As an example of training in the Remote World, we describe the remote instruction of a piano tremolo performance.

Beginners and experts use their arm muscles differently when playing a tremolo on the piano; that is, beginners focus on finger movement, while experts focus on wrist rotation. Focusing on this difference in muscle activity, we are developing a technology that uses EMS to directly transfer the way an expert uses their muscles to the muscles of a beginner. We confirmed that the body can directly learn how skilled pianists efficiently move their bodies in a manner that enables them to play with less unnecessary effort in their forearms [2].

With this technology, we developed a conceptual system for training in the Remote World. A simple motion sensor is attached to the skilled pianist’s forearm to measure the rotational motion of the wrist and transmit it to the system. An EMS device is connected to the arm of a beginner, and the EMS is applied to alternate rotations of the wrists as the skilled pianist moves. Experienced players can be taught through audio and video as well as directly experiencing—through the system—the playing techniques of expert pianists in a manner that creates a new teaching and learning experience for both instructor and student. We are focusing on the coordinated movement of multiple muscles during a performance and investigating a system that allows multiple muscles to move in a manner coordinated through EMS. This system will help beginners play smoothly and better.

By applying the various sensing and intervention technologies we are developing for our motor-skill-transfer technology, we will continue to create new forms of education and training in the Remote-World era for specific cases such as daily movements, sports, and musical-instrument performances.

3. Initiatives concerning embodied-knowledge technology

We are researching the extraction and sharing of embodied knowledge (i.e., skills) that cannot be acquired through verbal understanding to (i) clarify the mechanism of acquisition of embodied knowledge in sports and (ii) establish technology for supporting people in remote areas in acquiring embodied knowledge possessed by others.

The acquisition of embodied knowledge is considered the ability to (i) modify the way one moves one’s body (muscles and bones) on the basis of the unique sensations that arise in one’s body while performing a certain physical action and (ii) perform the action more skillfully and appropriately (Fig. 2). However, the unique sensation of the body is confined to the person who performed the physical action, and it is difficult to directly capture and communicate that subjective sensation to others. In this initiative, using windsurfing as a subject, our embodied-knowledge technology extracts and shares the unique experience of professional athletes during high-performance sports by capturing their physical activities and the behavior of the natural environment and tools (i.e., state information) and reproducing those factors to form a similar experience and acquire the embodied knowledge of others.

Fig. 2. Acquisition of embodied knowledge.

This technology extracts embodied knowledge in four steps: (1) sensing video and behavior data of actual athletes during competition; (2) conducting retrospective interviews with the obtained data about the intention and awareness of physical actions; (3) identifying situations in which unique physical sensations occur during high-performance activities; and (4) combining information on the state of the body at that time with media such as onomatopoeia and models that express the physical sensations.

We aim to improve extraction accuracy by evaluating whether the athletes actually experience and are aware of the extracted information during actual competitions and providing feedback (Fig. 3). We are improving the means of extracting more effective information to support the acquisition of embodied knowledge by sharing the state information and media that form each unique experience among athletes and extracting common and different shared media by comparing them.

Fig. 3. Extraction of athlete-specific experiences.

We have begun developing this technology by constructing a bodily-sensory-reproduction simulator to reproduce state information in a manner that reproduces the unique sensations that others have on the basis of the extracted information (Fig. 4). This simulator is used to form unique sensations that others have by replaying in tandem the use of windsurfing equipment by a professional windsurfer, wind (natural environment), and video the windsurfer watched. We believe that this simulator will make it possible to share subjective experiences, which are normally difficult to share and “closed” in the individual, as experiences for other people. Professional windsurfers are currently using this simulator to evaluate it and provide feedback. Through these efforts, we are striving on a daily basis to improve extraction and sharing technologies with the goal of acquiring embodied knowledge to achieve, for example, speeds of up to 60 km/h or more in future windsurfing competitions. We will continue to study this technology with an eye on expanding it to non-sports fields in which skills involving physical movements are used.

Fig. 4. Bodily-sensory-reproduction simulator for sharing the unique experience of an athlete.

4. Collaboration with NTT DOCOMO Human Augmentation Platform

We are promoting efforts to enhance human capabilities in the Remote World in collaboration with the Human Augmentation Platform^®, which is a platform for enabling human augmentation that NTT DOCOMO has identified as one of the new values to be provided in the sixth-generation mobile communications system (6G) era as a means of extending human senses via networks [3, 4]. We believe that our goal of enhancing human capabilities through networks can be effectively reached by linking with NTT DOCOMO’s network technology, which transmits and extends human senses through networks.

At docomo Open House’22 (January 17–19, 2022), for visualizing the state of collaboration with NTT DOCOMO’s Human Augmentation Platform, we presented two exhibits. One was a demonstration of our motor-skill-transfer technology, which involved active bioacoustic sensing [5] of hand and finger shapes to reproduce another human hand or robot, and a demonstration of a piano tremolo performance using the movements of an expert. The other exhibit on our embodied-knowledge technology showed our bodily-sensory reproduction simulator that allows beginners and amateurs to experience the skills of a professional windsurfer (Fig. 5). We will promote (i) collaboration with NTT DOCOMO to implement real-time augmentation of human ability by taking advantage of ultra-low latency and other features of 6G and (ii) investigation of more-valuable technologies for the Remote World in collaboration with more external partners.

Fig. 5. Overview of the exhibition in collaboration with the Human Augmentation Platform (NTT DOCOMO).

5. Future developments

To achieve the Remote World, we will focus on the objective and subjective aspects of augmentation of human ability as a new user experience, especially in terms of transferring and sharing skills and physical techniques used during exercise regardless of time and place. We will combine these technologies and collaborate with external partners to provide even more valuable technologies.

References

[1]	Y. Aono, H. Seshimo, N. Matsumura, Y. Koike, and M. Matsumura, “Cybernetics Technologies for Symbiosis between People and Machines,” NTT Technical Review, Vol. 19, No. 12, pp. 36–41, Dec. 2021. https://doi.org/10.53829/ntr202112fa4
[2]	A. Niijima, T. Takeda, K. Tanaka, R. Aoki, and Y. Koike, “Reducing Muscle Activity when Playing Tremolo by Using Electrical Muscle Stimulation to Learn Efficient Motor Skills,” Proc. of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Vol. 5, No. 3, Article no. 123, Sept. 2021. https://doi.org/10.1145/3478110
[3]	Press release issued by NTT DOCOMO on Jan. 17, 2022 (in Japanese). https://www.docomo.ne.jp/info/news_release/2022/01/17_00.html
[4]	H. Ishikawa, “What Is the Human Augmentation Platform that Shares Human Movements and Senses,” ITU Journal, Vol. 52, No. 7, pp. 16–18, July 2022 (in Japanese).
[5]	Y. Kubo, “Fine-grained Hand-posture Recognition for Natural User-interface Technologies,” NTT Technical Review, Vol. 19, No. 3, pp. 37–39, Mar. 2021. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr202103fa7.html