 |
|||||||||||||||
|
|
|||||||||||||||
|
Feature Articles: Digital Twin Computing for Advanced Interaction between the Real World and Cyberspace Vol. 18, No. 9, pp. 31–35, Sept. 2020. https://doi.org/10.53829/ntr202009fa4 Approaches to Cyber-physical Interactions Linking the Real World and CyberspaceAbstractFrom among the technologies incorporated in the cyber-physical interaction layer connecting the real world and cyberspace towards achieving Digital Twin Computing, this article introduces biological-signal-decoding and feedback technologies to extend motor-function capabilities by feeding the results of operations done with a person’s digital twin back to him/herself and zero latency media technology that enables operations that do not feel uncomfortable by feeding back a prediction presentation that eliminates delay and overcoming the physical distance of the real world and human perception of time. Keywords: cyber-physical interaction, zero latency, biological signal decoding and feedback  1. Cyber-physical interactions linking the real world and cyberspaceNTT laboratories are aiming to actualize Digital Twin Computing (DTC) to create diverse virtual societies through interactions between various objects and humans at a higher level beyond the constraints of the real world and to expand the scope and potential of human activities by fusing virtual societies with the real world. From among the technologies incorporated in the cyber-physical interaction layer connecting the real world and cyberspace, this article introduces biological-signal-decoding and feedback technologies to extend motor-function capabilities by feeding the results of operations done with a person’s digital twin back to him/herself and zero latency media technology to have these operations be discomfort-free through feedback of near-future prediction results (Fig. 1).
2. Biological-signal-decoding and feedback technologies to make DTC possibleOne aspect of a human digital twin in DTC is that it is an interface for one’s real-world self to interact with various digital twins in cyberspace or interact with other real-world humans. Specifically, a digital twin enables experiences in cyberspace to be conveyed to one’s real-world self via a human-machine interface (HMI). Thus, an HMI plays an important role in the interactions between digital twins and the real world [1]. Digital twins of humans are not just human reproductions but also expand capabilities through exchange and fusion in cyberspace, which are key digital twin operations [1]. An HMI also plays an important role in enabling a human to demonstrate such capabilities in the real world. Therefore, we have embarked on HMI-technology (biological-signal-decoding* and feedback technologies) initiatives to achieve this from the perspective of expanded motor-function capabilities, which are particularly important elements of expanded human capabilities in both cyberspace and the real world. In general, motor control in the human body results when stimulation received by a sensory organ is perceived by the brain via the nervous system, cognitive activity such as planning for action occurs in the brain, and commands for the movements to be executed are transmitted via the nervous system to the musculoskeletal system where the muscles that received the signals contract [2]. The exchange and fusion between digital twins regarding sensory functions, nervous functions, musculoskeletal functions, the biological signals transmitted between these functions, and feedback of the results of these to humans in the real world to enhance functionality and signal distribution is our objective for enhanced motor-function capabilities. This initiative will make it possible for humans in the real world to acquire enhanced motor-function capabilities through physical support or training in which the motor-control functions and biological signals of a skilled person are projected onto a trainee. To feed expanded motor-function capabilities back to humans in the real world to enhance human motor functions, we believe the following biological-signal-decoding and feedback technologies are important.
In particular, we consider the role of the mediation functions to be important in expanding capabilities through the digital twin operations of exchange and fusion. Exchanging and fusing the functions and signals of human motor control between digital twins will not only involve simple feedback to the real world but will also require detection and adjustment for inconsistencies between the real world and digital twin information expected to occur with such feedback. More specifically, it will be necessary to absorb the individual differences in functionality and signaling and the differences in the states of digital twins and humans in the real world. For example, if biological signals from a skilled athlete are fed back to someone who is unskilled, the unskilled person might not be able to move as originally expected due to their muscle strength, limited range of movement of their joints, or their current posture. There are also safety issues that may lead to ruptured muscle fibers, joint injuries, etc. Therefore, we are aiming to develop HMI technology that includes mediation functions in which the strength of signals to be fed back are adjusted or converted for movement more suitable to the human to create a world in which people with extended capabilities move as if with their own body. We are proceeding with research and development of these biological-signal-decoding and feedback technologies to apply exchange and fusion, which are central to DTC, and extend human capabilities.
3. Zero latency media technology for DTCIn the real world, the concepts of space and time can be considered closely linked as individual people and objects are defined by unique spatial coordinates and the time axis. Information exchanges are carried out at the speed of light for visual information or the speed of sound for auditory information. Therefore, information transmission cannot physically exceed the speed of light. For this reason, delays occur when operating something remotely, which means physical constraints must be transcended to recognize the object being operated as being in the same space. Humans also experience cognitive delays with information transmission when signals are sent to the brain from sensory organs. Recognition and judgment are then carried out, leading to movement, etc. For this reason, the cognitive constraints of human recognition and judgment must be surpassed, even for events that take place in the same space. By focusing on physical latency that occurs with communications, etc. when operating objects and latency due to human cognition through DTC, we have embarked on the zero latency media technology initiative with a focus on the perspectives of distant-independent expanded capabilities that transcend the constraints of spatial distance and individual-independent expanded capabilities that step into the world of cognition not dependent on personal abilities. Below, we give an overview of these expanded capabilities and introduce the direction in which we would like to take them. 3.1 Distant-independent expanded capabilitiesIn DTC, in which people and objects are fused and coordinated when providing feedback to the real world through the cyber-physical interaction layer, the concept of space-time must be defined in the cyberspace framework instead of the simple synchronous time progression in the digital-world presentation layer. There is always a physical communications delay when providing feedback to the real world, which can cause problems with a range of applications. Examples include discomfort due to lip-sync deviation with video streaming [3] or video motion sickness that occurs due to delays in system processing with virtual reality and augmented reality [4, 5]. These issues manifest as discomforts that occur when sense-organ information different from the information expected by the human is given [4], which is also a major problem with DTC. As well as feedback to the real world, there are even greater delays in human-to-thing and human-to-human interactions between remote locations, which necessitate technology to align the two different time axes to be fused. For example, in principle, with signal processing to remotely operate a machine, predicting environmental changes occurring with respect to the input makes alignment to the same time axis in the same space-time in cyberspace possible. However, simple physical prediction alone does not lead to a world where people and objects are fused. It is known that perceptual discrepancies about operations occur when people manipulate objects, but these can be eliminated with training and experience. Taking the example of driving a car, different capabilities are used when driving at 50 km/h and when racing at 300 km/h to process the sensation of driving gained from handling the vehicle or the resistance between the tires and the road surface, and the cognitive sense of time is completely different. Hence, we are conducting research into modeling how prediction is performed in the brain as well as prediction of environmental changes by matching the cognitive delays (called sensory delays) that arise with human body movements for manipulating objects with system problems that occur when operating objects and delays in remote communications (called physical delays). The aim is to expand capabilities to freely operate objects remotely by eliminating the discomfort that occurs due to these physical and sensory delays and overcoming, through cyberspace, the physical gaps in distance in the real world. 3.2 Individual-independent expanded capabilitiesCognitive-delay tolerances and predictive abilities that occur when humans manipulate objects vary greatly from person to person. For example, there are large differences in the abilities of skilled workers, such as surgeons, airplane pilots, car racers, or professional athletes, compared to those of untrained humans, and it is possible to eliminate such differences in cognitive ability by experience and training. Although acquiring some of these skills requires a great deal of training, providing a simulated experience of the world experienced by skilled workers is also an effective way to learn. For example, learning to pilot an aircraft or drive a racing car does not start out with maneuvering an actual machine but entails sufficient training with a flight or drive simulator. Similarly, training for competitive swimming is possible by having the experience of swimming fast by being pulled by a rope, or baseball training to raise one’s ability to respond to fastballs can be done by practicing with high-speed balls from a pitching machine. There has also been research into expanded capabilities that involves providing people with the experience of improved reflexes by stimulating the muscles using electrical muscle stimulation [6], enabling responses faster than the human cognitive delay. This research showed that it is possible to vary the amount of stimulation to give participants the illusion that they are actually performing a behavior such as grabbing a falling stick or taking a picture of a flying ball. In light of this, we will study the human ability to make predictions by researching both human physical-reflection mechanisms and prediction mechanisms by modeling the speculative decision-making that takes place in the brain. By making it possible to exchange these prediction capabilities, we also aim to expand individual-independent capabilities to make it possible to exceed one’s limitations and those in human capabilities in general. 3.3 The world we aim for and its applicationThrough expanded distance-independent capabilities and individual-independent capabilities, we aim to achieve a world in which remote objects can be operated freely by manipulating space-time in cyberspace to overcome the physical distance in the real world. This could entail remotely operating a device such as a robot arm in space or on the other side of the world in real time with the sensations of grabbing, moving, or crafting with the human hand. We will also work on extended capability technologies in DTC by studying semi-autonomous control technologies required for the negative latency world beyond zero through prediction considering the interactions between people and objects in their surrounding remote environments. References
|
|||||||||||||||
