Feature Articles: 2021 International Sporting Event and NTT R&D—Technologies for Preparing the Event

Vol. 20, No. 2, pp. 55–59, Feb. 2022. https://doi.org/10.53829/ntr202202fa7

Venue-management Staff × CUzo

Takahiro Kusabuka, Yuichi Maki, Takuya Gouda,
Akira Suzuki, Masaya Tamaru, and Takuya Indo

Abstract

For the 2021 international sporting event in Tokyo, NTT brought out a feature-distribution communication technology called CUzo and a mobile device with a transparent display called CUzo Card. This article describes our work on this project, which sought to enhance the hospitality afforded at venues by providing venue staff with information and communication technology tools to support the guidance they provide to visitors, and the technology used in this project.

Keywords: CUzo, transparent displays, AR

1. Overview

The purpose of this project was to improve the hospitality provided at venues by using information and communication technology (ICT) to support the efforts of venue-management staff who provide guidance to visitors including athletes and spectators. It is commonplace for people to use smartphones and tablets when dealing with inquiries, but they tend to do so while constantly looking at the screen of their device. This makes it difficult to properly engage in communication with other people while understanding their intentions based on non-verbal cues such as facial expressions and gestures. For this project, we used a system consisting of a feature-distribution communication technology called CUzo and handheld device called CUzo Card with a transparent display based on this technology (Fig. 1, Table 1) with which we implemented a communication style whereby venue-management staff could provide visitors with guidance while observing visitor expressions.

Fig. 1. A CUzo Card.

Table 1. CUzo Card specifications.

In 2021, this system was used at the international sporting event in Tokyo and was provided with applications to support face-to-face translation (Fig. 2) and facility guidance (Fig. 3). In face-to-face translation, the CUzo Card displays the translated content of a conversation in a manner similar to subtitles, allowing two people to have a conversation while watching the other person’s facial expressions and gestures at the same time. When guiding visitors at a facility, this system can also be used to add visual annotations to the user’s actual surroundings. However, the event had to be held without spectators due to the COVID-19 pandemic, but the system was nevertheless used by approximately 250 venue-management staff to support attendees including event officials and athletes at three venues in Tokyo and the surrounding regions for a total of 28 days.

Fig. 2. Using the face-to-face translation app.

Fig. 3. Using the facility guidance app.

2. Applications

2.1 Facility guidance

When venue-management staff are showing visitors the way to their seats and on-site facilities, they can provide clear guidance by holding up the CUzo Card to provide annotations on the visitor’s actual surroundings, including the location of the facility and the route to their destination.

Previously, it would have been necessary to go through the complicated procedure of searching for the target facility on the basis of the information confirmed from a flat map while comparing this information with the actual scenery then rebuilding a mental picture of the route. Some venues do not have signs or facilities that can serve as landmarks, and even if explanations are attempted using descriptive words such as “the white sign,” this can still lead to difficulties where smooth guidance is not possible when both parties understand these words as referring to different landmarks.

CUzo can solve this problem by superimposing annotations on the user’s actual surroundings so that they can be seamlessly guided through the facility. In this project, we prepared guides to show people to their seats at sports events, and to other facilities such as restaurants, stores, toilets, multi-purpose (barrier-free) toilets, prayer rooms, lounges, medical offices, and automated external defibrillators.

2.2 Face-to-face translation

Although smartphone translation functions can be used to handle multiple languages, these functions are not geared toward face-to-face conversations, but it is assumed that both parties in a conversation are speaking while checking their smartphones for the translated results. As a result, they end up being glued their phones and more likely overlook the other person’s facial expressions and gestures, which are regarded in conventional customer service as being important for understanding what someone is really trying to say. To address this issue, we designed an interactive system that allows users to view information while looking at the other person’s facial expressions through a transparent display. This system is used by holding up the display between the two participants in a conversation. When one of them speaks, their words are translated into subtitles that are displayed where the other person can see the speaker’s facial expressions and gestures together with the content of their speech (Fig. 4(a)). This system supported eight languages: Japanese, English, Spanish, French, Portuguese, Chinese (traditional and simplified), and Korean.

Due to the COVID-19 pandemic, it was felt that people might be unwilling to engage in face-to-face conversations, so we prepared the screen design shown in Fig. 4(b) to enable people to converse in a side-to-side manner without having to flip the screen around. Users can switch between the two screen designs at the press of a button, allowing the system to respond flexibly.

Fig. 4. Screenshot of the face-to-face translation app.

3. CUzo feature-distribution communication technology and the CUzo Card

The CUzo feature-distribution communication technology allows high-performance services to be accessed through simple devices by using communication to link up with distributed functions deployed on a network. By keeping the devices simple, we were able to address issues that can degrade the user experience, such as lack of responsivity when the device is subject to increased processing loads, and we were also able to improve the device’s battery lifetime and reduce initial deployment cost. The CUzo Card is a compact and lightweight device based on CUzo, incorporating the latest transparent display technology. Since the CUzo Card has a transparent display, users can view information from both sides of the display while looking through it at their actual surroundings.

4. System

The system we provided for this project is shown in Fig. 5. Venue-management staff carry CUzo Cards on neck straps and can assist visitors by communicating with them via facility guidance and face-to-face translation apps. The app server runs on the cloud, and the CUzo Card connects to this server via a smartphone over a Bluetooth^® connection.

Fig. 5. System configuration.

5. Results

Since the international sporting event went ahead in 2021 without any spectators, only the face-to-face translation function was used. However, the CUzo Card was widely used for guiding foreign athletes and staff. Over a 28-day period, approximately 3500 translations were carried out by approximately 250 venue-management staff members at three venues. The system received favorable responses from users, who not only thought it looked cool but also appreciated being able to see who they were speaking to. This was due to the attractive and convenient design of the device based around a transparent display, providing users with a tool that allowed them to converse more naturally with other people while observing their expressions.

Users also found the device easy to operate and use, confirming the utility of our user-experience design of enabling people to use the system without disrupting actual work situations. However, some venue-management staff members who used the system in direct sunlight said that it was sometimes difficult to see the screen, so we will need to consider the characteristics of the display when used outdoors.

Regarding the translation function, some users said they were glad to be able to access other language pairs besides Japanese and English, and the system was welcomed by users who had few staff members capable of speaking languages such as French or Spanish. It was also pointed out that the speech-recognition results were sometimes inaccurate, and that the translated results sometimes took quite a while to appear, highlighting the need for improvements to the speech-recognition/translation engine performance and the bandwidth of the local network through which the devices were connected.

6. Summary

By applying our CUzo feature-distribution communication technology and our CUzo Card device with a transparent display for providing guidance for visitors to venues used during the 2021 international sporting event, we were able to provide a system that allows this sort of work to be performed more smoothly. This system is expected to be useful in a wide variety of customer-facing scenarios. Examples include public transport facilities such as train stations and airports, public amenities, and tourist facilities. Although in this case we asked the venue-management staff to use the system, the fact that they found it so easy to use suggests that CUzo Card could also be carried by visitors and tourists while travelling or when visiting places such as art galleries and museums.

In the future, we intend to continue with our ICT research and development to explore new experiences at places where people and information come into contact.

	Takahiro Kusabuka Research Engineer, NTT Human Informatics Laboratories. He received an M.S. in engineering from Ritsumeikan University, Shiga, in 2012 and joined NTT in the same year. His research interests include mainly user interface and human-computer interaction. In this project, he engaged in launching the project and is currently the tech lead for the development of CUzo Card. He is a member of the Association for Computing Machinery (ACM).
	Yuichi Maki Research Engineer, NTT Human Informatics Laboratories. He received a Bachelor of Engineering and Master’s of Information Sciences from Tohoku University, Miyagi, in 2009 and 2011. He joined NTT in 2011 and has been engaging in research and development (R&D) of human-computer interaction, mobile computing, and consumer electronics. In 2019, he joined the 2020 Epoc-Making Project, where he engaged in the application development of the CUzo mobile transparent display for event venue staffs.
	Takuya Gouda Research Engineer, NTT Human Informatics Laboratories. He joined NTT DATA in 2009 and has been engaged in system development for the financial industry. He was in charge of using CUzo for staff of the 2021 international sporting event held in Tokyo.
	Akira Suzuki Senior Research Engineer, NTT Human Informatics Laboratories. He received a B.E. from Gunma University in 1989. Since joining NTT in 1989, he has been involved in R&D of facsimile communication, system development of communication terminals, and development of communication services. He participated in the Epoch-Making Project from 2019 to 2021 and was responsible for the application development and operation of CUzo for the international sporting event held in Tokyo in 2021.
	Masaya Tamaru Senior Research Engineer, NTT Human Informatics Laboratories. He received a B.E. from Nihon University, Tokyo, in 1997 and joined NTT the same year, where he has been engaging in R&D of home gateway, system development of home appliance control, Internet of Things services development for agriculture, etc. He participated in the Epoch-Making Project from 2019 to 2021 and was the development leader of CUzo for the international sporting event held in Tokyo in 2021. He is currently working for NTT EAST as an associate manager, Office of Network Security Network Business Headquarters.
	Takuya Indo Senior Research Engineer, Supervisor, NTT Human Informatics Laboratories. He received a B.S. in physics from Kyoto University in 1993. In the same year, he joined NTT. His research interests include media processing and human-computer interaction. He is a member of the Information Processing Society of Japan, and ACM.

↑ TOP