Feature Articles: Olympic and Paralympic Games Tokyo 2020 and NTT R&D—Technologies that Colored Tokyo 2020 Games

Vol. 20, No. 1, pp. 59–66, Jan. 2022. https://doi.org/10.53829/ntr202201fa10

Torch Relay Commemorative Photography × Ultra-realistic Communication Technology Kirari!

Yoshiyuki Mihara, Masami Nagata, Keisuke Hasegawa,
Taiji Nakamura, Junji Watanabe, Tatsuya Matsui,
Hideaki Iwamoto, and Shingo Kinoshita

Keywords: Olympic and Paralympic Games, Torch Relay, ultra-realistic communication technology Kirari!

1. Overview

To foster the momentum of the Olympic Games Tokyo 2020, we wanted as many people as possible to experience the Olympic Torch Relay. We therefore created three experiential exhibits that enabled participants to take commemorative photos with a torch as a personal experience of the Olympic Torch Relay. The first exhibit was the Virtual Torch Kiss, Torch Relay Commemorative Photo Corner: Kirari! Version, where participants could carry an actual torch and virtually experience a torch kiss using the Kirari! ultra-realistic communication technology. Kirari!, which conventionally needs to be set up in a laboratory and include large devices, was packaged into a truck, making it possible to hold the exhibit anywhere around the country. The second exhibit was the Torch Relay Commemorative Photo Corner: Optical Illusion Version. By projecting a special light that elicits phantasmagoric optical illusion onto a still image, we created the mysterious effect of making a supposedly unmoving image of a flame appear to move. Unlike shooting in front of a regular panel, it was possible to provide a more dynamic experience despite using still images. The third exhibit was the Torch Relay Commemorative Photo Corner: Kabuki Version. This exhibit combined kabuki, which has a long history in Japan, and the Olympic Torch Relay, which connected the regions of Japan, by enabling participants to take a commemorative photo with a torch while having kabuki makeup virtually applied to their faces.

2. Torch Relay Commemorative Photo Corner: Kirari! Version

We modified a truck that enabled participants to do a virtual torch kiss with the professional tennis player Kei Nishikori, who was holographically projected using Kirari! inside the truck (Fig. 1). The holographic display was based on a two-dimensional (2D) aerial image display [1]. We set up the display on the floor of the truck and mounted a transparent screen at a 45-degree angle from the display. The image of Kei Nishikori shown on the display was reflected and transmitted to the transparent screen, making it appear as if he were standing on the stage.

Fig. 1. Kirari! truck exterior view.

Setting up the stage for this purpose usually requires time for installing truss parts and setting the transparent screen diagonally. For this celebration event, since we had to move and install the transparent screen on the truck every day, we implemented a structure that made it easy to store and install the transparent screen. Thus, we were able to complete the stage setup, which would usually take a day, in only about two hours. Because also we had to travel throughout the country for 121 days, we needed to display the exhibit in a bright environment during the day, depending on the actual venue and the time of the event. Therefore, we used a light-emitting diode (LED) panel instead of a projector to improve the brightness of the original image and enhance outdoor visibility. We installed these pieces of equipment on the loading deck of a 10-ton truck.

2.1 Flow of experience

Participants first climbed onto the stage on the truck’s loading deck from the rear entrance of the vehicle and received a torch from the staff. We also installed a lift to enable wheelchair users to experience the exhibit.

Once the participant bearing the torch was in the proper position, a holographic image of Kei Nishikori appeared on the stage in such a way that the virtual Kei Nishikori and real-life participant appear to be standing side-by-side (Fig. 2). We prepared two patterns for displaying the holographic image of Kei Nishikori: one where he was standing up and one where he was kneeling. This made it possible for participants to make a torch kiss regardless of their height.

Fig. 2. Torch Relay Commemorative Photo Corner: Kirari! Version experience.

Since participants could not see Kei Nishikori directly on the transparent screen, we prepared a separate monitor through which the participants could view the movement of Kei Nishikori’s torch as they moved their torch toward his torch and do a virtual torch kiss. The torch carried by the participant was equipped with an infrared sensor to detect its position. As the participant’s torch approached Kei Nishikori’s, its tip lit up with an image of the Olympic flame.

The experience was captured using a camera installed off stage, and the video was instantly uploaded to an Internet server. We provided a means for issuing a 2D code of the video’s URL (uniform resource locator) to the participant after the experience so that they could immediately view the recorded video on their smartphone and post it on social networking services (SNS).

2.2 Results

The novel coronavirus (COVID-19) continued to spread during the Olympic Torch Relay period, and many municipalities canceled local torch relay celebration events, so the number of exhibitions was less than originally planned. We were able to hold the torch commemorative photo corner using the Kirari! truck at 34 venues in 25 prefectures, enabling a total of 4498 people to experience the exhibit. Many people were surprised at the realness of the life-size holographic image of Kei Nishikori appearing on stage, with many participants saying that they were able to gain a realistic experience of being an Olympic torchbearer.

To improve the realness of the Torch Relay experience, we added a function to simulate the tactile sensation of the fire being lit. This allowed people with visual impairments to also enjoy the experience (Fig. 3). Since the torch kiss experience with Kei Nishikori’s holographic image was visual based, visually impaired individuals had no way of knowing when and how it was done. Therefore, by adding a function that vibrated the torch simultaneously with the torch kiss, we were able to deliver a haptic experience for the passing of the flame from Kei Nishikori’s torch to the participant’s.

Fig. 3. Visually impaired person trying the virtual torch kiss experience.

This was done by attaching a small vibration device, which can be switched on and off by an external infrared signal, to the torch, and having a staff member turn the vibration on and off in accordance with the timing of the torch kiss for visually impaired individuals. We also designed the vibration to make it easier for visually impaired individuals to sense flames. This experience was provided only at one venue (Yokohama, June 30, 2021). A visually impaired person who experienced it commented that “Although the vibration was weak immediately after being lit, it gradually became stronger as I raised the torch above me, so I was impressed by the effort put into making it possible to feel even the changing intensity of the fire.”

3. Torch Relay Commemorative Photo Corner: Optical Illusion Version

In this exhibit, a special light was projected onto a panel showing an image of Kei Nishikori bearing a torch to trigger the illusion that the flames on the panel and the flurry of cherry blossoms in the background are moving (Fig. 4). Position matching was important for this exhibit because light was applied on the feature points on the panel. To accurately carry out alignment at each venue, we accurately projected the light to a specific location on the panel by linking the projector with the camera. This made it possible for the local staff to easily set up the exhibit.

Fig. 4. Torch Relay Commemorative Photo Corner: Optical Illusion Version experience.

A total of 5616 people in 3373 groups experienced this exhibit. All participants were surprised at the movement of still images that were not supposed to move.

4. Torch Relay Commemorative Photo Corner: Kabuki Version

NTT began its initiatives to create new experiences by combining traditional culture with information and communication technology (ICT) in 2016. As part of these initiatives, we have been focusing on kumadori, the distinctive kabuki style of makeup, and created the unusual experience of having your chosen kumadori mask virtually adhere to your face using ICT. Packaging this experience exhibit in a container, we have traveled to Fukushima and Kumamoto prefectures to provide a fun experience through ICT. Building on these initiatives, we provided the Torch Relay Commemorative Photo Corner: Kabuki Version as an experience with the theme of “connecting,” as exemplified by the Olympic Torch Relay that connected the regions (Fig. 5).

Fig. 5. Torch Relay Commemorative Photo Corner: Kabuki Version booth.

4.1 Flow of experience

First, participants chose their preferred mask from among the many painted with kumadori makeup (Fig. 6). We also projected the same special light used in the optical illusion version to make it appear that the kumadori masks hanging on the wall were casually changing their expressions. Once the participant stood in front of the monitor and placed their chosen mask on their face, the system automatically recognized the kumadori mask and overlaid it on the face of the participant. The kumadori mask had augmented reality (AR) features, i.e., it followed the person’s facial expressions and movements in all directions on the monitor in real time while the mask remained attached to the person’s face. The background in the monitor was also rendered to match the selected kumadori mask. We also prepared a background specifically inspired by the Olympic Torch Relay (Fig. 7). With their chosen mask attached to their faces, participants could pose any way they wanted while carrying the torch. Going to the next room, participants then found their faces being projected onto a gigantic 1.2-meter-high, three-dimensional (3D) face, to give them an element of surprise (Fig. 8). Photos of the experience were later posted on SNS and made available for download.

Fig. 6. Selection of kumadori mask.

Fig. 7. Torch Relay Commemorative Photo Corner: Kabuki Version experience.

Fig. 8. A gigantic 3D face with a height of 1.2 m.

4.2 Results

This experience was intended to be offered at the Olympic Torch Relay expanded celebration events at two locations, namely, the Expo ’70 Commemorative Park in Osaka and the Yokohama Red Brick Warehouse Square in Yokohama. However, due to the spread of COVID-19, it was only held at the Yokohama Red Brick Warehouse Square. A total of 82 people, including Olympic torchbearers and other people involved, experienced the exhibit. Participants laughed once they saw their chosen kumadori masks attached to their faces upon holding the masks up, and many got into character as kabuki actors while taking commemorative photos with the torch in hand.

5. Summary

Due to the spread of COVID-19, the number of people who experienced the exhibits was less than planned. Nevertheless, we were able to deliver surprises and smiles to the participants at the venues where we could provide the experiences. We believe that these interactions with the advanced technologies of our laboratories, which are not normally accessible to the public, enabled many people to experience firsthand NTT’s innovativeness.

Going forward, we will continue to work on creating new experiences using ICT and connect people with each other.

Acknowledgments

We would like to express our gratitude to the members of the Tokyo Organising Committee of the Olympic and Paralympic Games and the representatives of the local governments who were involved in the operations. We also would like to thank the members who handled the operation of the Kirari! truck at the celebration venues.

NTT is an Olympic and Paralympic Games Tokyo 2020 Gold Partner (Telecommunication Services).

Reference

[1]	S. Uchida, E. Ashikaga, M. Imoto, M. Wagatsuma, and K. Hidaka, “Concept of Immersive Telepresence Technology ‘Kirari!’,” Proc. of the Media Computing Conference 2015, p. 39, 2015 (in Japanese).

	Yoshiyuki Mihara Senior Research Engineer, NTT Human Informatics Laboratories. He received a B.Sc. and M.Sc. from Tokyo Institute of Technology in 2004 and 2006. He also received a Ph.D. from Kyoto University in 2017. Since joining NTT in 2006, he has been engaged in research and development (R&D) of a home network management service. The home network management protocols he designed have been standardized by Universal Plug and Play (UPnP) Forum, International Telecommunication Union Telecommunication Sector (ITU-T), and Japan’s Telecommunication Technology Committee (TTC). He is currently promoting the key protocols with a view to launching a home network management service.
	Masami Nagata Manager, NTT 2020 Public Relations Strategic Business Development Division. She joined NTT DOCOMO in 2002 and has been engaged in marketing. She is currently responsible for asset activation of the Tokyo 2020 Olympic Torch Relay as the Presenting Partner.
	Keisuke Hasegawa Engineer, NTT Human Informatics Laboratories. He received a B.E. and M.E. in informatics from Kyoto University in 2012 and 2014. He joined NTT WEST in 2014 and engaged in the maintenance of network facilities. Since joining NTT Service Evolution Laboratories in 2016, he has been engaged in research of media processing for the ultra-realistic communication technology Kirari!. He is a member of the Institute of Image Information and Television Engineers (ITE).
	Taiji Nakamura Senior Research Engineer, 2020 Epoch-Making Project, NTT Human Informatics Laboratories. He joined NTT DATA Communications Systems (now NTT DATA) in 1991. He has over 20 years’ experience in information systems planning and development in the national public sector.
	Junji Watanabe Senior Distinguished Research Scientist, NTT Communication Science Laboratories. He received a Ph.D. in information science and technology from the University of Tokyo in 2005. His research is focused on cognitive science and haptic communication devices with applied perception. His academic work has been published in scientific journals in the field of psychology and interface technologies. He has also presented his work at technology showcases and art festivals such as SIGGRAPH and Ars Electronica. His current research interest is how technologies are designed to achieve the wellbeing of society.
	Tatsuya Matsui Senior Research Engineer, NTT Information Network Laboratory Group. He received a B.Sc. and M.Sc. from Tokyo Institute of Technology in 1993 and 1995. Since joining NTT in 1995, he has been engaged in R&D of the cyber security, digital library, MICE (meetings, incentives, conventions, exhibitions) application, interactive exhibition “Henshin Kabuki.” He is currently working to improve the working style of researchers at NTT labs.
	Hideaki Iwamoto Research Engineer, NTT Human Informatics Laboratories. He received a B.E. and M.E. in information engineering from Kyushu Institute of Technology, Fukuoka, in 1991 and 1993. He joined NTT Information and Communication Systems Laboratories the same year and engaged in R&D on information retrieval and natural language communication. He is currently researching and developing innovative user interface/user experience design and its showcasing.
	Shingo Kinoshita Vice President, Head of NTT Human Informatics Laboratories. He received a B.E. from Osaka University in 1991 and M.Sc. with Distinction in technology management from University College London, UK, in 2007. He joined NTT in 1991 and was a senior manager of the R&D planning section of the NTT holding company from 2012 to 2015. He is currently a visiting professor at the Art Science Department, Osaka University of Arts, and visiting executive researcher at Dentsu Lab Tokyo. He has served as a member of the Japan Science and Technology Agency (JST) JST-Mirai Program Steering Committee, member of the All Japan Confederation of Creativity (ACC) TOKYO CREATIVITY AWARDS 2021 Judging Committee, and member of the Broadband Wireless Forum Steering Committee. He has been engaged in R&D of a media-processing technology, user interface/user experience, communication protocols, information security, machine learning, service design, and technology management. Until recently, he had been in charge of NTT’s Tokyo2020 initiatives, including sports-watching video technology, inclusive design for social issues, and promoting the use of ICT in kabuki, entertainment, and media arts such as live music. He has been in his current position since 2021, where he manages R&D on information and communication processing of humans based on human-centered principles.

↑ TOP