Feature Articles: Olympic and Paralympic Games Tokyo 2020 and NTT R&D—Technologies for Viewing Tokyo 2020 Games

Vol. 19, No. 12, pp. 78–84, Dec. 2021. https://doi.org/10.53829/ntr202112fa10

Marathon × Ultra-low-latency Communication Technology

Soichiro Usui, Shinji Fukatsu, Eiichiro Matsumoto,
Maiko Imoto, Daisuke Shirai, and Shingo Kinoshita

Abstract

NTT provided ultra-low-latency communication technology to the Tokyo 2020 Real-time Remote-cheering Project, which was implemented by the Tokyo Organising Committee of the Olympic and Paralympic Games for the marathons of the Olympic and Paralympic Games Tokyo 2020 held in Sapporo on August 7 and 8, 2021. This project proposed a new means of watching sports by connecting the marathon course in Sapporo with the cheering venue in Tokyo in real time. By delivering the cheers of spectators to the athletes from the remote location in Tokyo, it was possible to create a sense of unity between the athletes and spectators, as well as a sense of realism similar to roadside cheering while maintaining safety and security. Our efforts concerning this project and the technologies that we used are introduced in this article.

Keywords: ultra-low-latency communication technology, remote cheering, IOWN

1. Overview of the project

During the novel coronavirus (COVID-19) pandemic, it has become increasingly difficult to cheer on athletes directly at competition venues. By using new communication technologies, the Tokyo 2020 Real-time Remote-cheering Project aimed to create a world in which athletes at a competition venue and spectators in a remote area can share the excitement, emotion, and sense of unity at the venue (Fig. 1).

Fig. 1. Overview of the Tokyo 2020 Real-time Remote-cheering Project.

As a Gold Partner of the Olympic and Paralympic Games Tokyo 2020, NTT has been contributing to the creation of the most-innovative Games in history by providing cutting-edge communication technology. In light of the situation concerning the COVID-19 pandemic, we adopted a new mission to propose a new form of sports viewing using communication services and make it a springboard for the future and decided to provide the technology for this project.

Regarding remote spectator experience, delay time is the largest challenge in transmitting cheering from remote venues to the competition venue. For marathons in particular, even a small delay can have a significant effect on the ability to transmit cheering to athletes running at 5 m/s. The total delay in watching a sporting event remotely had been several seconds each way, which includes not only the propagation delay of light but also media-processing delay such as transmission-processing delay and compression delay for video information; as a result, it has been impossible to transmit cheering to players, athletes, etc. without a delay. NTT undertook the Tokyo 2020 Real-time Remote-cheering Project with the goal of solving these problems by using its ultra-low-latency communication technology to minimize the latency of the transmission process to about 100 ms each way and ensure the transmission of cheering to the athletes.

2. Device configuration of the project

The overall device configuration of the project is shown in Fig. 2. We aimed to create a space in which remote spectators could feel as if they were cheering along the roadside as the runners were running by. Therefore, we installed light-emitting diode (LED) displays (approximately 50 m wide and 2 m high) at both the Sapporo marathon course (in front of Sapporo Sosei Square) and Tokyo remote-viewing venue. The displays enabled the runners and spectators to see each other in actual size. The total width of the LED displays at both sites was approximately 50 m, but the location and spacing of the displays varied in accordance with the conditions at the installation site (Figs. 3(a) and (b)). Eight cameras were set up in front of the LED displays at both sites to capture images of spectators and runners (Fig. 3(c)). The 4K images from each of the two sites were transmitted to the other site via a 20-Gbit/s high-speed broadband line via a media-processing device using ultra-low-latency media-processing technology and transmission device using disaggregated configuration for video transport. These two technologies constitute our ultra-low-latency communication technology.

Fig. 2. Device configuration of the project.

Fig. 3. The setup at both venues.

3. Ultra-low-latency communication technology

The Innovative Optical and Wireless Network (IOWN) is an innovative concept proposed by NTT as a next-generation communication platform. IOWN will enable the creation of a network and information-processing infrastructure that includes devices capable of providing high-speed, high-capacity communications and vast computing resources by using innovative technologies centered on optical technology. One of the components of IOWN, the All-Photonics Network (APN), will achieve ultra-low-power consumption, ultra-high capacity, and ultra-low latency by introducing photonics-based technology into everything from the network to devices. The ultra-low-latency communication technology used in this project uses two APN elemental technologies: disaggregated configuration for video transport and ultra-low-latency media-processing technology (Fig. 4).

Fig. 4. NTT network equipment that provided ultra-low-latency communication technology.

3.1 Disaggregated configuration for video transport

Disaggregated configuration for video transport enables flexible changes in configuration, addition of functions, and cost reduction by separating network functions that were previously provided in combination and configuring them so that each function can be controlled by standardized interfaces. We developed a plug-in unit (video PIU) in which a direct serial digital interface (SDI)-signal acquisition function is added to a white-box switch with an optical transponder for long-distance transmission. The new video PIU enables direct transmission of uncompressed video and audio in SMPTE ST 2110^* format over optical long-distance transmission lines. As a result, the delay between the video input on the transmitter side and video output on the receiver side was reduced to about 1 ms, and the one-way delay (including the distance delay between Tokyo and Sapporo) was reduced to about 20 ms. We also implemented a set of functions for video routing in the network operating system, which is the control software for the white-box switch, to achieve integrated operation of network and video devices (Fig. 5).

Fig. 5. Disaggregated configuration for video transport.

3.2 Ultra-low-latency media-processing technology

Ultra-low-latency media-processing technology captures SDI video signals output from a camera on a subframe or line basis and executes video processing such as geometric transformation and composition at the video-signal level. Implementing each video process at the video-signal level instead of at the frame level makes it possible to reduce the frame latency required by conventional video processing and achieve low latency (Fig. 6).

Fig. 6. Ultra-low-latency media-processing technology.

In this project, when transmitting video from multiple cameras, the SDI video signals from each camera were made into subframes in line units, and the received subframes were immediately separated and displayed on multiple displays in a manner that achieved even lower latency (Fig. 7).

Fig. 7. Aggregation and disaggregated processing of multiple camera images.

*	SMPTE ST 2110: A standard developed by the society of motion picture and television engineers (SMPTE) for transmitting video over Internet Protocol (IP) networks for the professional media industries.

4. Results of the project

On the days of the men’s and women’s marathons, after a demonstration of low latency was conducted, the remote spectators cheered for the runners. Via the 50-m-wide display, the Tokyo venue provided a sense of realism to the spectators, namely, the feeling that the athletes were running right in front of them, and the spectators could show their support by clapping and waving flags. For past remote-cheering experiences, due to delays, the timing of the cheering was not synchronized with the athletes, so the spectators found it difficult to feel a sense of unity while cheering for the athletes. For the Sapporo marathon course, however, the ultra-low-latency communication technology enabled the spectators to see their cheering reaching the runners as the runners passed by. As a result, the spectators were able to feel the same sense of unity even in the Tokyo venue, which made them even more enthusiastic about cheering (Figs. 8(a) and (b)).

Through this cheering experience, the spectators at the Tokyo venue commented, “I could experience the same sense of realism and speed as cheering at the roadside,” “I felt that I could cheer from a position closer to the athletes than possible at the roadside, which is usually crowded,” and “Above all, it was lovely to see the cheering reach the athletes.” In other words, the spectators seemed to be very satisfied with the sense of unity by having their cheering reach the marathon venue, an outcome that had been impossible with the remote cheering. The significance of this project was also confirmed by the fact that some of the marathon runners showed interest in the real-time cheering from the remote area via the display, and that result was a world’s first. As expected, the delay time for the transmission process was kept to 100 ms each way. Unfortunately, typhoon Mirinae was approaching Japan, and the men’s marathon was held in rainy weather on August 8th; nevertheless, after ensuring safety of the athletes and spectators, we were able to transmit cheering from Tokyo to Sapporo in spite of the rain (Figs. 8(c) and (d)).

Fig. 8. Scenes from the project during the marathons.

5. Future developments

In light of the results of this project, NTT aims to create a new style of watching sports competitions that allows people who are unable to cheer at a competition venue to experience the sense of presence and unity, which could only be experienced at the venue, in a safe and secure manner. Regarding the Tokyo 2020 Real-time Remote-cheering Project, the cheering was transmitted from a special venue in Tokyo. We will accelerate our research and development so that our ultra-low-latency communication technology can be widely used and the same real-time remote cheering will be possible from homes.

NTT is an Olympic and Paralympic Games Tokyo 2020 Gold Partner (Telecommunication Services).

	Soichiro Usui Senior Manager, Strategic Business Creation Team, Research and Development Planning Department, NTT. He received a B.S. in industrial engineering and management from Tokyo Institute of Technology in 1999. He joined NTT the same year and was involved in corporate business and network services development at NTT EAST. He was with NTT Research and Development Planning Department from 2012 to September 2021, where he was in charge of information and communication technology (ICT) business creation for the entertainment and sports domain. He has been with NTT EAST since October 2021.
	Shinji Fukatsu Senior Research Engineer, Supervisor, Cyber-World Laboratory, NTT Human Informatics Laboratories. He received a Ph.D. in engineering from Osaka University in 2002 and joined NTT the same year. He has been engaged in research and development (R&D) of human interfaces and video streaming services as well as in planning and development of video streaming services at NTT Plala and in promoting standardization and international development of ICT at the Ministry of Internal Affairs and Communications. He is currently engaged in R&D of remote-world infrastructure technology.
	Eiichiro Matsumoto Senior Research Engineer, Cyber-World Laboratory, NTT Human Informatics Laboratories. He received a B.E. in science engineering (information science) from Meiji University in 1999 and joined NTT the same year, from August he was engaged in the development of geographic information systems and voice/video communication systems at NTT EAST. From 2008 to 2010, he was involved in the R&D of IP television (TV)/mobile broadcasting standards and the formulation of technical standards at NTT. Since 2010, he has been engaged in the planning and development of network services and systems for video distribution systems, such as IPTV and radio-frequency TV and researching video-transmission-system technologies for adding high value to relay networks and access networks at NTT EAST. He has been at his current position since 2020.
	Maiko Imoto Research Engineer, Cyber-World Laboratory, NTT Human Informatics Laboratories. She received an M.S. in computer science from Ochanomizu University, Tokyo, in 2011. She joined NTT the same year and began studying geographical information search for mobile phones and HTML5-based e-learning applications. She was involved in R&D on Kirari! immersive telepresence technology. From 2017 to 2020, she was responsible for the development of live-video distribution platforms for broadcasters and creating live viewing services for sports and entertainment at NTT Plala. She has been engaged in R&D of multimedia data interactive transmission technologies.
	Daisuke Shirai Senior Research Engineer, Supervisor, Frontier Communication Laboratory, NTT Network Innovation Laboratories. He received a B.E. in electronic engineering, M.E. in computer science, and Ph.D. in media design from Keio University, Kanagawa, in 1999, 2001, and 2014. He pioneered the world’s first 4K JPEG 2000 codec system, which enables low latency 4K60p video transmission on a Gigabit network. He has applied his expertise across multiple domains through his study of practical applications in digital audio and video broadcasting technology, image coding, information theory, networking, human-computer interaction, and software architecture. His current research topics include remote video production networks, ultra-low latency visual communication, and its security over optical transport networks.
	Shingo Kinoshita Vice President, Head of NTT Human Informatics Laboratories. He received a B.E. from Osaka University in 1991 and M.Sc. with Distinction in technology management from University College London, UK, in 2007. He joined NTT in 1991 and was a senior manager of the R&D planning section of the NTT holding company from 2012 to 2015. He is currently a visiting professor at the Art Science Department, Osaka University of Arts, and visiting executive researcher at Dentsu Lab Tokyo. He has served as a member of the Japan Science and Technology Agency (JST) JST-Mirai Program Steering Committee, member of the All Japan Confederation of Creativity (ACC) TOKYO CREATIVITY AWARDS 2021 Judging Committee, and member of the Broadband Wireless Forum Steering Committee. He has been engaged in R&D of a media-processing technology, user interface/user experience, communication protocols, information security, machine learning, service design, and technology management. Until recently, he had been in charge of NTT’s Tokyo2020 initiatives, including sports-watching video technology, inclusive design for social issues, and promoting the use of ICT in kabuki, entertainment, and media arts such as live music. He has been in his current position since 2021, where he manages R&D on information and communication processing of humans based on human-centered principles.

↑ TOP