Remote Robot Control with Haptic Feedback Enabled by Low-latency Transport and Precision Bilateral Control Technologies

2.1 Low-latency transport technology

NTT is advancing the development of the All-Photonics Network (APN) as part of its commitment to implementing the Innovative Optical and Wireless Network (IOWN)^*1, a next-generation communication technology. In APN IOWN1.0, the network can provide a stable, low-latency physical infrastructure by exclusively dedicating optical wavelengths from end-to-end, crucial for mission-critical services. However, it is essential to extend this low latency and stability to robot-side applications. To fully leverage the capabilities of the IOWN APN, our research and development (R&D) efforts are focused on developing low-latency transport technologies that ensure high-quality data delivery directly to the application layer.

2.1.1 Uncompressed video transmission technology [1]

Uncompressed video transmission technology is a method that enables low-latency feedback of visual information from remote robots to operators. With the recent increase in network capacities, although the required transmission bandwidth becomes larger compared with buffering and compressing video frames, the ability to transmit video with low latency has become more advantageous in real scenarios.

Traditional optical transmission devices were offered as integrated solutions by individual manufacturers, including both hardware and software functions, which made it difficult for service providers to flexibly add additional functionalities for video transmission. In contrast, new open optical transmission devices have started to emerge with a disaggregated configuration. These devices allow for the separation of various functions and the addition of flexible configurations and enhancements through standardized interface control. We have seized this trend and developed an uncompressed video transmission plug-in unit (VideoPIU) that can be used with disaggregated open optical transmission devices (Fig. 2).

Fig. 2. Uncompressed video transmission plug-in unit (VideoPIU) and specifications.

VideoPIU converts serial digital interface (SDI) signals^*2 directly into SMPTE ST2110 streams [2], the standard for Internet protocol (IP)-based video transmission, allowing direct video-signal transmission through open optical transmission devices. VideoPIU is also implemented in hardware, reducing the delay from video input on the transmitting side to video output on the receiving side to within 1 ms. It can process 8K60p video^*3 per image and, by linking two images, can achieve 8K120p transmission. It also supports seamless protection (SMPTE ST2022-7 [3]), ensuring continuous delivery without interruption even in the event of network disruptions or temporary failures by using two different optical paths.

2.1.2 Remote direct memory access acceleration technology

Remote direct memory access (RDMA) communication is widely used in datacenters and high-performance computing domains as a method for achieving high-bandwidth and low-latency data access between computers. RDMA allows for the direct transfer of data from memory to the network without central processing unit (CPU) intervention, making it a highly promising technology for fast, low-latency data transfers essential for time-sensitive and real-time services.

However, RDMA reliable connection, which facilitates reliable data transfers, is primarily designed for short-distance communications within datacenters. This design limitation has historically led to performance issues when adapted for medium to long-distance communications. The latency induced by longer distances typically results in increased waiting times at the requester due to slower acknowledgement (ACK) reception from the responder, subsequently lowering throughput.

To address this issue, we devised a method for generating pseudo-ACKs near the requester, allowing for the early release of the requester’s work queue element and enabling the acceptance of subsequent work requests. This method helps prevent performance degradation by reducing wait times and maintaining throughput (Fig. 3(a)).

This innovative method places accelerators near both the requester and responder, bridging long-distance networks. These accelerators create pseudo-ACKs using connection establishment data and request packets, effectively minimizing the waiting period for transmissions. However, this initially resulted in disabled packet-loss detection and retransmission mechanisms, posing significant reliability concerns. To resolve these, we implemented a suitable retransmission mechanism for the communication segments (Fig. 3(b)). This mechanism uses the Go-Back-N and selective retransmission strategies by segment, enabling quick recovery from packet loss while maintaining high-speed communication over long distances.

Fig. 3. Technical highlights of RDMA accelerator.

We conducted simulation evaluations of the above method. In the evaluations, we simulated propagation delays typical of long-distance networks and compared the throughput with traditional RDMA communications. The results indicate that under conditions simulating a 1000-km delay, with and without packet loss, there was an average improvement in throughput of 10 and 40 times, respectively, for a message size of 4 KiB (Fig. 4).

Fig. 4. Effect of accelerators in RDMA communications.

2.2 Precision bilateral control technology

We have sought collaboration with Sony, which is developing precision bilateral control technology. Sony possesses advanced bilateral control technology capable of flexibly adapting to changes in the external environment in response to human operations and controlling very minute forces with extreme accuracy. This section outlines the overview of the precision bilateral control technology.

Precision bilateral control technology is achieved by synchronizing the actions of two robots: a leader and follower. The follower robot moves in response to the leader robot’s actions, and when the follower robot touches an object, the reactive force is transmitted back to the leader robot [4] (Fig. 5(a)).

2.2.1 High-sensitivity force sensing technology [5]

To feedback the sensation of touching human soft tissues to the operator, it is necessary to detect minute force changes as small as 1 gf (gram-force, equivalent to 0.0098 N) at the tool tip. However, incorporating force sensors into the system resulted in inertial forces being observed as noise, which made it challenging to detect slight changes in tip force. To address this issue, Sony has applied an optical strain sensor called a fiber Bragg grating (FBG) sensor, which features a diffraction grating etched into part of an optical fiber. Despite its ultra-fine fiber shape, the FBG sensor can measure the strain in the sensor part with high sensitivity. By mounting an ultra-sensitive FBG sensor at the tool tip and improving the algorithm that estimates the three-dimensional force applied to the tool tip from the sensor’s strain amount, Sony has significantly reduced dynamic noise. This advancement allows for the precise detection of even minute force changes as small as 1 gf (Fig. 5(b)).

Fig. 5. Precision bilateral control technology.

2.2.2 Precision position and force control technology

To achieve precise bilateral control, precise management of both position and force is essential. Traditional algorithms are affected by modeling errors due to variations in the robot’s posture, leading to degraded position-tracking performance and vibrations when interacting with external environments. To address this issue, Sony has incorporated its proprietary technology, the generalized inverse dynamics (GID) library^*4, into their systems. This integration minimizes the impact of modeling errors caused by changes in robot posture, significantly enhancing responsiveness and operational stability, achieving position accuracy of less than 1 mm.

High-speed real-time processing is crucial for the signal processing involved in precise bilateral control. These signal processing tasks have traditionally been implemented on generic CPUs within a host personal computer (PC), which could not meet control system specifications due to communication delays and interrupt processing delays. To overcome these challenges, Sony has implemented part of the electrical processing on a field programmable gate array (FPGA), which is suitable for high-speed parallel processing. A proprietary protocol has also been developed for communication between the host PC and FPGA, resulting in a system that is 50 times faster than previous systems. By cascading multiple FPGAs using optical fibers, it is now possible to transmit signals with minimal delay in multi-degree-of-freedom systems such as the precision bilateral control system composed of leader and follower robots.

3.1 Communication evaluation results

Throughout the long-distance demonstration experiment spanning 120 km via the APN, video was transmitted with a remarkably low total latency of 1.6 ms, broken down as 1 ms of video-processing delay and 0.6 ms of APN-transmission delay for the 120 km. For the bilateral control signal communication via RDMA, the total latency was maintained at 1 ms, with an accelerator-processing delay of 0.4 ms, APN-transmission delay of 0.6 ms, and jitter of only 10 µs. This setup allowed for exceptionally stable and low-jitter communication due to the elimination of CPU intervention, confirming the feasibility of achieving the high stability required by the bilateral control robots, as outlined in Table 1.

3.2 Operational evaluation of bilateral control robots

The operational evaluation revealed that the operator could not perceive any significant distance between themselves and the follower robot, enhancing the sense of immediacy and responsiveness. In this experiment, interactions with objects of varying hardness were performed remotely through the bilateral control robot, enabling the operator to discern even subtle differences in the object’s surface texture. The ELF-SR2 spatial-reproduction display also provided natural stereoscopic vision enabled by head tracking, allowing the operator to carry out natural manipulations with a clear sense of spatial depth.