Feature Articles: Dwango × NTT R&D Collaboration
Collaborative Trial: Application of HEVC Technology to Niconico Live
Dwango Co., Ltd. operates Niconico Live, the largest live streaming service in Japan based on the legacy video codec H.264/MPEG-4 (Moving Picture Experts Group, version 4) AVC (Advanced Video Coding). Meanwhile, NTT Media Intelligence Laboratories has been working on a high-speed software encoder based on H.265/MPEG-H, also known as HEVC (High Efficiency Video Coding), the updated high-performance successor to H.264. Dwango and NTT have now joined forces in a collaborative trial to test and evaluate the new HEVC video compression standard.
Keywords: live streaming, H.265/HEVC, real-time processing
1. Niconico Live
Niconico Live, run by Dwango Co., Ltd., is one of the largest live streaming services in Japan, averaging about 20 million accesses per day and 4 million visitors per month. The streaming content consists of Official Program run by the Dwango group, Channel Live run by other corporations or parties, and User Live by regular users that has become a popular feature. Another attractive feature of the service is that user comments are overlaid directly onto the video and synced to the playback, which enables viewers and the uploader to react to comments and engage in back-and-forth interaction.
This means that an encoder for the Niconico Live service must not only provide real-time video compression, it must also provide low-delay performance enabling back-and-forth interaction among users and be sufficiently economical that it sees widespread acceptance among ordinary users. The encoder used by the current service is based on the H.264/ MPEG-4 (Moving Picture Experts Group, version 4) AVC (Advanced Video Coding) standard and has been widely distributed to users under the name Niconico Live Encoder.
2. Collaborative trials using HEVC
Enhanced video compression efficiency is critically important to improve Niconico Live picture quality and to further expand the user base and scale of the service. Not surprisingly, Dwango took notice when the new international video coding standard H.265/MPEG-H, also known as HEVC (High Efficiency Video Coding), was approved by the ITU-T (International Telecommunication Union, Telecommunication Standardization Sector) in April 2013. HEVC markedly improves the performance of various encoding tools and effectively doubles the data compression ratio compared to H.264 at the same level of video quality. However, these performance gains sharply increase the compression processing load and thus require dedicated hardware for real-time compression. This hardware would be extremely costly for a live streaming service, especially for a service such as Niconico Live that offers many channels.
NTT laboratories have been working on the development of a software coding engine for HEVC high-speed compression for some time. This software encoder was specifically designed for 4K and other high-definition video formats for which there has been strong market demand, but repurposing the software for small-screen applications should make it possible to run the software in real time on a single personal computer (PC).
It is far more cost effective to build a live streaming system using software than using dedicated hardware. This is basically what motivated this collaborative project to customize NTT’s HEVC software encoder for Niconico Live, incorporate the software in Dwango’s system, and assess its performance when used to support a streaming service.
3. HEVC software encoder customization
For these trials, we needed to achieve real-time encoding of low-resolution video on a single PC for mobile devices, a starkly different direction from high-definition video encoding. We first customized the HEVC software encoder to enhance picture quality by tuning the bit allocation to place limited bits on critical portions of the image, and to enhance speed by customizing algorithms about parallel processing and mode decision. In particular, customizing the parallel processing algorithm was quite critical.
Just as in earlier encoding schemes, HEVC partitions single-image frames into blocks, then subjects each block to inter-/intra-picture prediction and transform coding. NTT’s software encoder features pipelining and therefore handles this processing in parallel with multiple central processing unit (CPU) cores at very high speed. With high-resolution 4K video, there is a very large number of blocks, so tasks allocated to each CPU core can be finely adjusted. This makes it fairly easy to increase CPU utilization. With low-resolution video, however, there are relatively few blocks, which results in processing load variations between the CPU cores. Needless to say, this reduces CPU utilization and can drag down the processing speed (see Fig. 1).
Since the current trial involves low-resolution video encoding, the encoder was customized by taking into account the number of blocks shared by each CPU core and the variation in load that might affect the block processing speed (see Fig. 2). Adoption of these measures means that the encoder can maintain a high level of CPU utilization even at low resolution, and low-resolution video is compressed by HEVC in real time on the CPU of the node PC.
4. Future developments
The next step is to conduct field trials. During the field trials, we will roll out the HEVC software encoder in the Niconico Live public channel and assess live streaming services delivered to smartphone users who are under especially tight network bandwidth constraints. If that goes well, we will extend the encoder deployment to other channels. This project is the largest real-time HEVC streaming trial so far, and it should provide a lot of valuable data on the effects of reduced bandwidth, QoS (quality of service), and a host of other technical matters. Through these trials, Dwango and NTT Media Intelligence Laboratories are demonstrating their ongoing commitment to enhancing the quality and development of new video streaming services.