Direction for Supporting Torchbearers × Swarm Communication Control Technology

1. Overview

NTT is working to create new experiences by combining art and technology, and one of the results of our research in this area is “Art of Swarms,” a spatial-expression technology using display bots. Spatial expression refers to the creation of an organically changing visual space through the coordinated above-ground movements of a swarm of robots equipped with multiple displays. Unlike projection through the entire floor surface, display bots, which have physical mass, move as a swarm to create a visual space that enables an unconventional and tangible visual experience.

NTT introduced the concept of spatial expression at the 2018 NTT R&D Forum. Since then, we have expanded the scale and examined the theatrical effects of using display bots, including through demonstrations at the world’s largest art festival in Linz, Austria in 2018 and exhibits at the Sports Viewing Re-Imagined Exhibition in July 2019. At the Sports Viewing Re-Imagined Exhibition, we presented a collaboration of a swarm of more than 30 display bots with prominent artists Akiko Nakayama [1] and Ei Wada [2]. We also used a swarm of display bots for sports expression, which received high praise from many people who watched the exhibit.

Using the knowledge gained from these demonstration experiments, we applied Art of Swarms to the Tokyo 2020 Olympic Torch Relay. We wanted many people to have a more dramatic experience of the Torch Relay and enable them to witness the possibilities of the fusion of NTT’s communication technologies with art. We directed a festival to welcome the torchbearers arriving at the venue during the NTT Presents Tokyo 2020 Olympic Torch Relay Celebration events (Figs. 1 and 2).

Fig. 1. Torch kiss scene at the Osaka event.

Fig. 2. Torch kiss scene at the Yokohama event.

2. System configuration and technologies used

Display bots are robots that move on the ground and equipped with hexagonal displays (Fig. 3). They are composed of a display component at the top and a driving function component at the bottom, each operating independently. This configuration enables the robots to move in any direction while keeping the display surface in a fixed direction, thereby increasing the breadth of the spatial-image expression. In addition to the display, the top part includes devices for communicating, positioning, and controlling the movement of the display bot. The bottom part includes the driving mechanism and battery.

Fig. 3. Display bots.

Figure 4 shows an overview of the entire system, including the display bots. The editing system is used for generating scenarios in which movements of multiple display bots and video-content assignments are programmed on the basis of the desired final image expression for the entire space. The control system then delivers and executes the specific scenarios and video content to the display bots. This enables display bots to move organically as a swarm.

Fig. 4. Overview of system configuration.

The position of each display bot must be measured accurately to enable coordinated movements while they move close to each other at a distance of around 10 centimeters. We had used motion capture with infrared cameras and reflective markers for positioning during previous demonstration experiments since they were conducted indoors. The same method, however, could not be applied since the torch relay event was going to be held outdoors. Although Global Positioning System (GPS) is commonly used for outdoor environments, the normal positioning accuracy of GPS exceeds a few meters, making it impossible to apply it as is. We therefore adopted a high-precision positioning system called real-time kinematic global navigation satellite system (RTK-GNSS). However, RTK-GNSS had not been widely used, and using it for simultaneous control of 60 robots had never been attempted. RTK-GNSS enables high accuracy (error of a few centimeters) even outdoors by using correction information generated in real time from observation data at different locations called base stations in addition to the positional information obtained from regular GPS antennas. For this project, we applied NTT DOCOMO’s “DOCOMO IoT high-precision GNSS positional information service” [3] to RTK-GNSS, a first-of-its kind initiative. Using RTK-GNSS enables accurate location information to be obtained, but it does not enable measurement of the exact direction. If the exact direction could not be determined, the direction of each display bot’s display would be different from one other, and the entire image would be become corrupt. We therefore installed two antennas diagonally on the display bot and obtained the direction information using the differential information from the two antennas.

We used proprietary standard Laird communications (2.4 GHz) and Wi-Fi communications (2.4/5.2 GHz) for communication between the display bots and control system and fourth-generation (4G) Long-Term Evolution (LTE) communications via mobile routers for communication with the DOCOMO IoT high-precision GNSS positional information service.

Before the display bots start moving, the scenarios are communicated simultaneously to each bot via radio signals from the control system upon the operator’s command. Each display bot then begins to move in accordance with the transmitted scenarios. The bots move in accordance with the programmed scenario, but they can also be controlled in real time by the operator, such as to stop display bots simultaneously or decrease their speed, since they always operate in concert with the control system.

3. Testing

Testing with 100 display bots requires a space that is large enough to store, charge, and operate them and that is relatively free from obstructions to enable reception of the satellite signals for RTK-GNSS. We therefore conducted tests at the NTT Tsukuba R&D Center (Tsukuba City, Ibaraki Prefecture), which meets these requirements. After the decision to postpone the Tokyo 2020 Games, at the time of test resumption, the plan was to hold the event in collaboration with members from Ars Electronica. However, COVID-19 infections did not subside as we had expected, increasing the likelihood that they could not come on the day of the event. We therefore promptly changed our policies as follows (Figs. 5 and 6).

• Control and preparations for actual event and tests at Tsukuba was carried out by only Japanese members mainly from NTT.
• Ars Electronica members did not come to Japan but supported the operations online.

Fig. 5. Remote collaboration with members of Ars Electronica.

Fig. 6. Testing at NTT Tsukuba R&D Center.

As a result, NTT members had to deal with many unfamiliar tasks, such as handling hardware issues, troubleshooting and analyzing problems, and other tasks that were originally supposed to be handled by members of Ars Electronica. Many hours were therefore spent on remote correspondence with members in Austria. Due to the time difference with Austria, there were many time constraints; thus, it was necessary to efficiently carry out testing, problem analysis, and response.

4. Live performance

We held the first live event in front of the Tower of the Sun Plaza at the Expo ’70 Commemorative Park on April 13, 2021. The final torch kiss with the last torchbearer of the day would have been welcomed through a spatial-expression show by a swarm of display bots, with the Tower of the Sun, the symbol of the Expo ’70 Commemorative Park, in the background.

Around April 7, when we traveled to Osaka for the preparations, the number of new COVID-19 infections in Osaka Prefecture began to increase sharply, and the declaration of the third state of emergency became imminent. Eventually, on April 7, an announcement was made that the Torch Relay along public roads would be canceled and held at the Expo ’70 Commemorative Park. Although we were able to avoid having the event on April 13 canceled, it was held with no spectators. In addition, the Osaka prefectural government requested us to avoid presenting in front of the Tower of the Sun, which was illuminated in red after sunset during the state of emergency. To comply with this request, we needed to start the event before sunset. However, since the display bots create the magical spatial expression by moving as a swarm of images in the dark, holding the event before sunset meant that the presentation would lose much of its charm. Also, given that many display-bot hardware failures had occurred since we arrived in Osaka and that GPS reception in the field was poor and positioning failed during preliminary rehearsals, to be safe, we decided to carry out the live performance in Osaka with 48 display bots in a stationary state. During the day of the event, while it was raining heavily, the stationary display bots welcomed the torchbearers and rendered a coordinated display of lights along with the torch kiss.

After the completion of the Osaka event, we were faced with two major issues to be resolved in preparation for the Yokohama event scheduled on June 30. The first issue was the repair of the broken display bots. More than half of the bots had some sort of problem, so we needed to quickly carry out troubleshooting and repair to ensure that as many display bots as possible were operational before the next event. For the repair of the display bots, we determined which ones had to be returned to Austria for repair and which ones could be repaired in Japan. Considering all possible factors, including the number of units necessary for the entire production, number of days required for repair, and number of days required for testing, we created a repair plan and changed it in accordance with the daily situation. As a result, we were able to secure 48 operational display bots before the live event.

The second issue was the stabilization of their positioning. The location of GPS satellites on the day of the event and the surrounding environment had a significant impact on positioning. Even if we conducted prior testing in the same location and obtained good results, there was no assurance that we could obtain the same results during the actual event. We therefore used two methods to increase the reliability of positioning.

The first was improving the antenna-installation method. From our experience in Osaka, we found that raising the position of the antenna could lead to more stable positioning. However, installing the antennas too high leads to many issues, such as difficulty in fixing the antenna in place or an unsightly appearance, so it was necessary to find a height that balances functionality with aesthetics. After various tests, we determined that the optimum height was around 15 cm from the surface of the display. The second was achieving failsafe positioning. We needed to ensure that positioning would continue even if the RTK-GNSS positioning fails. To achieve this, we used an odometer to estimate the position and direction using the amount of rotation, etc. of the wheel of the display bots.

The second event was held at Yokohama Red Brick Warehouse on June 30, 2021. Due to the COVID-19 situation, like in Osaka, onsite viewing through general admission was canceled, and the event was conducted exclusively through online streaming. From rehearsals to the live performance, the display bots did not fail, and their positioning was stable and almost never failed. In the area between Buildings No. 1 and No. 2 of the Red Brick Warehouse, the display-bot program showcased a magical display of lights through coordinated movements of 48 display bots as the last torch kiss of the day was exchanged and the final torchbearer was sent off to ignite the cauldron (Fig. 7).

Fig. 7. Direction support for the Torch Relay in Yokohama.

5. Summary

Onsite viewing through general admission was canceled for both the Osaka and Yokohama events, preventing the general public from experiencing the display-bot presentation in person. Nevertheless, we consider it a major achievement to have enabled many people to enjoy via online streaming the presentation of lights and movements by the swarm of display bots around the torchbearers and for the show to be featured in various media events, including the opening ceremony presentation, as an iconic scene of the Tokyo 2020 Olympic Torch Relay.

Going forward, we will continue to study further possibilities of the fusion of art and technology.

Acknowledgments

The implementation of the display-bot torchbearer-support program was made possible with the cooperation of many people. We would like to express our sincere gratitude to the people of Osaka and Kanagawa Prefectures for supporting the events, members of Dentsu and Dentsu Live, members of Ars Electronica, and many others who were involved in the preliminary rehearsals, tests, as well as during the live performance.

NTT is an Olympic and Paralympic Games Tokyo 2020 Gold Partner (Telecommunication Services).

References

[1]	Akiko Nakayama, http://akiko.co.jp/akikoweb/
[2]	Ei Wada, https://eiwada.com
[3]	DOCOMO IoT high-precision GNSS positional information service (in Japanese), https://www.nttdocomo.co.jp/biz/service/highprecision_gnss_positioning/