Feature Articles: ICT Solutions Offered by NTT Group Companies
REALIVE360: Multi-angle Virtual-reality Video-streaming Service that Gives the Viewer a Realistic Feeling of Being in a Theater
While various events have been canceled or postponed due to the novel-coronavirus (COVID-19) pandemic, a new style of event called “online streaming of spectatorless events” is emerging. The NTT WEST Group has developed a multi-angle virtual-reality video-streaming service called REALIVE360 and began providing it as a service to enable people in remote rural areas to experience concerts and other events held in urban areas in a highly realistic manner. How to use REALIVE360, its main features, and its future development are introduced in this article.
Keywords: highly realistic, multi-angle viewing, 3D sound
1. Current status of live music and other events during the COVID-19 pandemic
The novel-coronavirus (COVID-19) pandemic has dramatically changed the event industry. Under the state of emergency imposed by the Japanese government to stop the spread of the virus in early 2020, almost all events and performances were canceled, postponed, or reduced in scale. Tokyo Olympic and Paralympic Games were postponed, and professional baseball and football leagues have been forced to postpone the opening of their seasons. Even after the state of emergency was lifted, music festivals, local festivals, and other events that were scheduled to be held in 2020 had been cancelled, postponed, or scaled back considerably. Clusters of COVID-19 infections linked to music-related events have been reported, so people’s self-restraint generated under the state of emergency is continuing; thus, live-music venues across Japan have been closing one after another. The results of a computer simulation showed that if physical distance is maintained in a theater to prevent the spread of COVID-19, the room will be “full” at 15% capacity. These results have attracted significant attention . Under these circumstances, the holding of events offline is not expected to return to “normal” anytime soon.
2. Online streaming of spectatorless events
Although it is difficult to hold events offline, a new style of event called “online streaming of spectatorless events” is emerging. The band Southern All Stars performed a live broadcast without spectators, and the number of people who purchased tickets reached about 180,000, which generated sales of an estimated 650 million yen (equivalent to about 6 million US dollars) . That number of people was more than ten times the capacity of the venue being live streamed (17,000 people), and it has made the potential of online streaming known to a wider audience.
The advantage of online streaming is that the unit price of tickets is set at 2000 to 3000 yen, which is significantly lower than regular music concerts, so people who are willing to pay that amount will purchase tickets. What’s more, if the event is held on a weekday evening, fans from afar who are unable to get to the event for various reasons can simply decide to watch it online. Reducing the price of live-streaming tickets would be thought to lead to a decrease in sales; however, it made it possible to sell tickets to people who were reluctant to buy tickets and fans who had given up watching live shows.
NTT WEST Group’s Smart Life team began developing the multi-angle virtual-reality (VR) video-streaming service called REALIVE360 in 2018 (Fig. 1). Initially, our slogan for developing this service was to “Eliminate the disparity in experiences and opportunities between urban and rural areas through the power of ICT (information and communication technology).” The majority of entertainment-related events and people mobilized for such events, including live music, is concentrated in urban areas, and it is a hurdle for music fans living in rural areas to spend time and money on transportation to attend a concert in urban areas. Moreover, many people, such as the disabled, seniors, and families with small children, who love music, theater, and sports have difficulty getting to venues. We developed REALIVE360 as a service that enables such people to experience—with a high level of realism—concerts and other events held in urban areas, even if they live in remote rural areas (Fig. 2). In addition to the streaming technology of NTT WEST Group, we have established a collaborative system in partnership with companies such as Alpha Code Inc. that handles VR filming and editing. At the end of 2019, we streamed a live performance of Momoiro Clover Z, a Japanese idol group, which attracted a lot of attention as the first artist to be streamed. After that, as mentioned in the introduction, online streaming of events became the mainstream due to the COVID-19 pandemic. We collaborated with the radio station FM802 to implement REALIVE360 VR ZONE, a project to support the music industry, which is suffering from a decline in audience numbers and cancellation of events due to the pandemic (Fig. 3). We have accumulated achievements such as the streaming of the online event J-WAVE Innovation World Festa and the stage version of the animated movie “For Whom the Alchemist Exists.”
3.1 Streaming 4K/8K high-quality 360° VR videos
The first feature of REALIVE360 is unprecedented streaming of 4K/8K high-quality 360° VR videos. Our unique technology eliminates the coarseness of image quality that was previously associated with 360° VR streaming. REALIVE360 uses the Panorama Super Engine, which was developed by NTT laboratories and commercialized by NTT TechnoCross, to stream 4K/8K high-quality VR video. 360° VR video requires 360° video data, including parts that are not visible at certain moments, so it requires a different means of thinking than what we apply to 16:9 video displayed on an ordinary flat-screen display. The 4K video data displayed on a flat screen are expressed in 4K resolution on that screen. For 4K 360° VR video, however, the video data (including the parts that are not visible at certain moments) are set to be in 4K resolution; therefore, the actual viewing area is expressed at a resolution less than 4K. As a result, when a 360° VR video is viewed, the screen will be blurrier than expected, despite the 4K resolution. If the resolution is increased to 8K to increase the image quality in the viewing range, the bandwidth required for the streaming network will increase, and the requirements for the viewing environment will be more difficult to meet. In other words, the viewer’s network environment must be sufficiently refined, and that requirement might significantly affect the expansion of the number of viewers. Given these conditions, we decided to adopt the Panorama Super Engine, which can stream high-resolution video in the viewing area only and low-resolution video in the other unseen areas. When the Panorama Super Engine is used, the required network bandwidth can be reduced by about 75% for 4K resolution and 85% for 8K resolution, allowing viewers to enjoy 360° VR videos with 8K resolution even over a 4G (fourth-generation) mobile connection (Fig. 4).
3.2 Multi-angle viewing
The second feature of REALIVE360 is multi-angle viewing. Since viewers can freely select the angle from which they want to watch from multiple angles, they can proactively and subjectively enjoy the live performance from the angle they prefer. The viewer can also select angles that would not normally be possible, such as following favorite band members, looking across the audience from the stage as the performers would do, or looking down from the ceiling. Because this function is multi-angle, the viewer can set an extraordinary sub-angle while setting the main angle that firmly holds the performer in a bust shot.
3.3 3D sound
The third feature of REALIVE360 is three-dimensional (3D) sound. For example, the singer’s voice can be heard from the right side of the image, while the drums can be heard from the back of the image. The sound heard from each actual angle can be realistically reproduced in the VR video. The way the sound is heard changes not only when the viewing angle changes but also when the point being viewed in the VR video changes. Therefore, we are trying to increase the sense of reality not only in the image but also in the audio (Fig. 5).
REALIVE360 service is provided via an application, so it is not necessary to wear special goggles or headsets; in other words, it allows the viewer to enjoy the world of VR by simply using a smartphone or tablet. What’s more, since REALIVE360 is linked with various ticket agencies, viewers can purchase tickets from one of those agencies, authenticate it with the issued serial code, and view the event via the REALIVE360 app.
4. Future developments
As explained above, REALIVE360 is currently mainly used for entertainment. We have received many requests for additional functions from event organizers who are service users, and we are currently developing the following functions; (i) those that enable the user to do online what used to be done when running events offline, such as a function that allows performers and viewers to communicate interactively and a function that displays advertisements such as sponsors; (ii) those that can be implemented only by online streaming, such as a money-transfer function that allows viewers to send money with a feeling of gratitude and support to the provider and a function for analyzing viewing data; and (iii) those that improve viewing extensibility, namely, a cast function allowing the VR videos to be viewed on a television or a web browser in addition to on the current app.
We plan to further expand REALIVE360 service into other fields such as education and tourism to cultivate the business-to-business market. We are also considering partnerships that transcend industry boundaries and hope to hear a wide range of opinions to help us solve problems and create new value in a variety of business situations.
Authors (from left): Masanori Emura, Manager, Business Design Department; Takahiko Sasahara, Senior Manager, Business Design Department; and Takafumi Fukatani, Chief, Business Design Department; NTT WEST Corporation