Cybernetics Technologies for Symbiosis between People and Machines

1. Toward symbiosis between people and machines

As the computational efficiency of processers has increased significantly, the accuracy of image processing and recognition has improved; technology that handles cross-modal information, such as various sensors, has been refined; and automated driving of cars, which until now has only been depicted in science fiction, is gradually being achieved. The improvement in voice recognition, character recognition, and context understanding makes it possible for automatic-response engines, such as chatbots, to support counter services (contact persons), and people now carry around mobile devices (such as smartphones) with high-performance processors. These factors have led to rapid establishment of an environment in which we can use advanced information and communication technology (ICT) services in our daily lives. One of the technologies that contributes significantly to improving the accuracy of recognition and understanding is artificial intelligence (AI), which is constructed by machine learning using large amounts of real data.

Current mainstream AI technology is akin to modeling logical and objective recognition and understanding that works like the human left brain based on our experience and memory, and it has enriched our lives mainly by targeting things that can be expressed in words and those that can be symbolized such as images and sensors data.

We aim to create a society that enables people and machines to coexist in harmony (Fig. 1) by updating conventional AI technology with the following two steps:

Fig. 1. Technologies we are researching.

2. Motion support

Cybernetics, proposed by Norbert Wiener in 1948, is a unified approach to communication and control concerning living organisms and machines. Paying renewed attention to this concept, we are developing technologies to support the motion of people.

Human motion is executed when motor instructions from the brain are transmitted to the muscles, which then contract on receiving the instructions. The brain then perceives and recognizes the result as a stimulus through sensory organs and repeats planning a new motion and sending its appropriate instruction. We are attempting to support this communication and control in the human body by sensing and feeding back biological signals such as electroencephalography (EEG) and electromyography (EMG).

More specifically, we aim to create a society in which people can move (live) better by reproducing (transcribing) their current motor skills temporarily and permanently on the basis of the motions of others (skilled people) and their former selves. For example, we are considering (i) training that uses electrical muscle stimulation (EMS) to reproduce the muscle motions of skilled people executing various skills and sports and (ii) rehabilitation based on records of one’s motions in daily activities.

We are working on enabling the following four functions, which we consider necessary for reproducing (transcribing) human motor skills by using biological signals (see Fig. 2).

• Measuring biological signals and constructing kinematic models: Bio-signals (EEG, EMG, etc.) of the transcription source are sensed and a kinematic model representing the relationship between the bio-signals and the motion is constructed.
• Generating and providing sensory stimulation: Sensory stimuli (EMS, vibration, exoskeleton, etc.) that control the motion of the transferee are generated and expressed on the basis of the generated kinematic model.
• Estimating dynamics: The motion results and biological signals of the transferee (which was stimulated and moved) are sensed, and states such as posture and muscle contraction are estimated.
• Adjusting sensory stimulation parameters: The parameters of sensory stimuli are adjusted to attain better motor results and natural motor experiences on the basis of the estimation results concerning the motion state and kinematic model. New feedback based on the results is then provided to the transferee.

Fig. 2. Efforts to reproduce (transcribe) human-motion ability by using biological signals.

We had been working on bio-sensing technology (using, for example, EMG) and feedback technology using tactile and EMS to the human body. Mechanisms of human motion, however, are complex, and various efforts are underway in the fields of physiology, brain science, and biomedical engineering to support and enhance motion, so an interdisciplinary approach is necessary. Looking to the future and using our accumulated technological strengths, we are planning to acquire new knowledge about the mechanisms of motion through creation of technology and interdisciplinary collaboration with internal and external experts. For a society in which people and machines coexist at a higher level, we will elucidate human processing and action processes related to motion and expand human capabilities.

3. Embodied knowledge

In human life and social activities, knowledge is the basis for improving and refining the quality and efficiency of people’s behavior and thinking. It is therefore important for people living in the present to acquire knowledge that has been accumulated over a long period and apply it in various situations in their daily lives.

The first thing that comes to mind when people think of acquiring knowledge is studying at school or learning a subject. Such already systematized and described knowledge is called explicit knowledge, and everyone has experienced the widely practiced methods of acquiring and demonstrating such knowledge. However, knowledge does not mean only the explicit type. Another type is the knowledge that people learn and express that is difficult to systematize or describe, and it can only be acquired through experience rather than through mental understanding. This type is what we call embodied knowledge.

Regarding learning to play a musical instrument, people generally refer to instructional books that systematically describe how to handle the instrument, its functions, and how to move the fingers. Therefore, if you read all of this instructional material and understand it, will you be able to play an instrument? Of course, the answer is “no.” In other words, only by moving your body (fingers, in this case) and practicing with that understanding in mind will you be able to play. This reality indicates the existence of embodied knowledge, which is the knowledge rooted in a physical activity, as opposed to explicit knowledge expressed in textbooks (Fig. 3).

Fig. 3. What is embodied knowledge?

Embodied knowledge is largely based on the senses and subjectivity of people, and it is far from being supported by ICT for acquiring and demonstrating in the manner of explicit knowledge, which is objective and easily verbalized. However, with the recent evolution of technology, it has become possible to sense and process information at high speed in a manner closer to the sensations received by the people involved. Accordingly, we are conducting research on embodied knowledge from the perspective of neo-cybernetics, which captures the mechanism of actions from the starting point of the person’s cognition. Taking windsurfing as a subject, we are clarifying the mechanism of acquiring and demonstrating embodied knowledge in sports and making it possible to support this mechanism. With this in mind, we are studying an analysis method to find the latent pattern between the physical activity of the athlete, behavior of the natural environment and tools, and performance of the athlete. Moreover, using the data acquired from actual competitors, we are also studying how to communicate the discovered pattern to other people and how to improve the value (acquisition and demonstration of embodied knowledge or changes in interest in the sport and the natural environment) provided to the recipients of the communication (Fig. 4).

Fig. 4. Embodied knowledge concerning windsurfing.

Although sports is typical in the study of embodied knowledge, the aforementioned skills, which are knowledge acquired and demonstrated in strong connection with physical actions, such as playing a musical instrument, drawing a picture, or making a good presentation, exist in a variety of everyday situations. Accordingly, we are aiming to create a society in which anyone can acquire and apply embodied knowledge, which has been confined within certain people, through ICT based on the perspective of the person who wants to acquire that knowledge while simultaneously studying areas other than sports.

4. Zero latency

Physical work by remotely operated robots is promising for solving the labor shortage in the service industry, such as nursing and caregiving, and for ensuring safety in dangerous work, such as at medical institutions and disaster sites, during a pandemic. However, such remote operation always involves a delay between the operator and robot, and that delay is known to have adverse effects such as work errors, reduced work efficiency, and operator fatigue. It can thus be said that while the sophistication of remotely operated robots has increased their practicality, the effects of delays are becoming impossible to ignore. The delay is caused by not only the communication but also the operation of the camera, display servo, etc., and the control of these systems. Even if a low-latency system is used and remotely controlled over a very short distance, a round-trip delay (namely, the sum of the delay between the incidence of light at the camera and screen display to the operator and the delay between the operator’s operation and actuator’s operation) of 100 ms or more occurs.

We therefore aim to develop a remote-operation system that can fully demonstrate the capabilities of the operator by eliminating the negative effects of delays through four technologies: prediction of the operator’s motions, prediction of physical changes in the work area, correction of the robot’s motions, and correction of the operator’s perception (Fig. 5).

Fig. 5. Four technologies for achieving zero latency.

One example of predicting the operator’s motion and correcting the robot’s motion is predicting the motion of the operator several hundred milliseconds in the future and moving the robot before the operator performs the motion in accordance with the prediction. Therefore, the effect of delay is reduced (Fig. 6). However, it is possible that the robot’s motion may be wrong (namely, unintended by the operator), so we are currently working on improving prediction accuracy by estimating the operator’s intentions on the basis of the content of the work and motion of objects.

Fig. 6. Experimental system for predicting the operator’s motion and correcting the robot’s motion.

As an example of predicting physical changes in the work area and correcting the robot’s motion, we simulated a robot’s physical structure to determine how it moves according to the operator’s motions. Presenting the simulation results to the operator through augmented reality (AR) and other means allows the operator to recognize the results of the robot’s motions before the robot actually moves, which reduces the effects of delays (Fig. 7). Since the operator will be working on the basis of AR-based predictive information, in contrast to direct work, we are currently studying how to present information to support the operator more effectively during various tasks.

Fig. 7. Experimental system for cognitive correction using AR.

5. Future developments

We are currently refining these technologies on an individual basis. In the future, we will unify these technologies to provide services especially for domains that require flexible and fuzzy human-oriented support.