Collaborative Trial: Application of HEVC Technology to Niconico Live

1. Niconico Live

Niconico Live, run by Dwango Co., Ltd., is one of the largest live streaming services in Japan, averaging about 20 million accesses per day and 4 million visitors per month. The streaming content consists of Official Program run by the Dwango group, Channel Live run by other corporations or parties, and User Live by regular users that has become a popular feature. Another attractive feature of the service is that user comments are overlaid directly onto the video and synced to the playback, which enables viewers and the uploader to react to comments and engage in back-and-forth interaction.

This means that an encoder for the Niconico Live service must not only provide real-time video compression, it must also provide low-delay performance enabling back-and-forth interaction among users and be sufficiently economical that it sees widespread acceptance among ordinary users. The encoder used by the current service is based on the H.264/ MPEG-4 (Moving Picture Experts Group, version 4) AVC (Advanced Video Coding) standard and has been widely distributed to users under the name Niconico Live Encoder.

2. Collaborative trials using HEVC

Enhanced video compression efficiency is critically important to improve Niconico Live picture quality and to further expand the user base and scale of the service. Not surprisingly, Dwango took notice when the new international video coding standard H.265/MPEG-H, also known as HEVC (High Efficiency Video Coding), was approved by the ITU-T (International Telecommunication Union, Telecommunication Standardization Sector) in April 2013. HEVC markedly improves the performance of various encoding tools and effectively doubles the data compression ratio compared to H.264 at the same level of video quality. However, these performance gains sharply increase the compression processing load and thus require dedicated hardware for real-time compression. This hardware would be extremely costly for a live streaming service, especially for a service such as Niconico Live that offers many channels.

NTT laboratories have been working on the development of a software coding engine for HEVC high-speed compression for some time. This software encoder was specifically designed for 4K and other high-definition video formats for which there has been strong market demand, but repurposing the software for small-screen applications should make it possible to run the software in real time on a single personal computer (PC).

It is far more cost effective to build a live streaming system using software than using dedicated hardware. This is basically what motivated this collaborative project to customize NTT’s HEVC software encoder for Niconico Live, incorporate the software in Dwango’s system, and assess its performance when used to support a streaming service.

3. HEVC software encoder customization

For these trials, we needed to achieve real-time encoding of low-resolution video on a single PC for mobile devices, a starkly different direction from high-definition video encoding. We first customized the HEVC software encoder to enhance picture quality by tuning the bit allocation to place limited bits on critical portions of the image, and to enhance speed by customizing algorithms about parallel processing and mode decision. In particular, customizing the parallel processing algorithm was quite critical.

Just as in earlier encoding schemes, HEVC partitions single-image frames into blocks, then subjects each block to inter-/intra-picture prediction and transform coding. NTT’s software encoder features pipelining and therefore handles this processing in parallel with multiple central processing unit (CPU) cores at very high speed. With high-resolution 4K video, there is a very large number of blocks, so tasks allocated to each CPU core can be finely adjusted. This makes it fairly easy to increase CPU utilization. With low-resolution video, however, there are relatively few blocks, which results in processing load variations between the CPU cores. Needless to say, this reduces CPU utilization and can drag down the processing speed (see Fig. 1).

Fig. 1. Reduction of CPU utilization.

Since the current trial involves low-resolution video encoding, the encoder was customized by taking into account the number of blocks shared by each CPU core and the variation in load that might affect the block processing speed (see Fig. 2). Adoption of these measures means that the encoder can maintain a high level of CPU utilization even at low resolution, and low-resolution video is compressed by HEVC in real time on the CPU of the node PC.

Fig. 2. Equalization of processing loads.

4. Future developments

The next step is to conduct field trials. During the field trials, we will roll out the HEVC software encoder in the Niconico Live public channel and assess live streaming services delivered to smartphone users who are under especially tight network bandwidth constraints. If that goes well, we will extend the encoder deployment to other channels. This project is the largest real-time HEVC streaming trial so far, and it should provide a lot of valuable data on the effects of reduced bandwidth, QoS (quality of service), and a host of other technical matters. Through these trials, Dwango and NTT Media Intelligence Laboratories are demonstrating their ongoing commitment to enhancing the quality and development of new video streaming services.

Ryuichi Tanida Research Engineer, Visual Media Coding Group, Visual Media Project, NTT Media Intelligence Laboratories. He received the B.E. and M.E. in electrical engineering from Kyoto University in 2001 and 2003, respectively. He joined NTT to study video coding in order to develop a real-time H.264 software encoder application. He moved to NTT Advanced Technology Corporation in 2009 to manage various software development projects related to video compression, distribution, and TV phones. He has been at NTT Media Intelligence Laboratories since 2012 as a research engineer in the visual media coding group. His current research focuses on the development of HEVC software encoders. He is a member of IEICE (Institute of Electronics, Information and Communication Engineers.

Kenichi Miyazaki Platform Division Manager, Dwango, Co., Ltd. He joined Dwango as a mid-career employee in December 2009. He has developed many Niconico services such as niconico-jikkyou*1 and niconico-denwa*2 as well as Niconico’s back-end systems. Since October 2014, he has been in charge of the development of the entire Niconico system. *1 niconico-jikkyou is a communication service enabling viewers to post comments on current broadcast services such as TV programs. *2 niconico-denwa is a communication service enabling live video distributors to talk on the phone with viewers. This service was terminated in Dec. 2014.