You need Adobe Reader 7.0 or later in order to read PDF files on this site.
If Adobe Reader is not installed on your computer, click the button below and go to the download site.

Feature Articles: Video Technology for 4K/8K Services with Ultrahigh Sense of Presence

Vol. 12, No. 5, pp. 37–41, May 2014. https://doi.org/10.53829/ntr201405fa7

QoE Assessment Methodologies for 4K Video Services

Kimiko Kawashima and Jun Okamoto

Abstract

The technological developments of 4K/8K video enable users to watch video content with high quality and a high sense of presence. To provide attractive 4K video content, it is important to design and manage services based on the viewers’ quality of experience (QoE). This article introduces QoE assessment methodologies and standardization activities.

Keywords: 4K video, QoE, subjective quality assessment

1. Introduction

The 4K and 8K video specifications have been attracting attention in recent years for their use in next-generation video services, and various electronics manufacturers are already selling 4K televisions (TVs). Also, the Next Generation Television & Broadcasting Promotion Forum (NexTV-F) established by the Ministry of Internal Affairs and Communications has begun studying standards and specifications for 4K video transmission and reception. This forum’s objective is to create an environment in which any user can enjoy an immersive high quality and high sense of presence video experience, which gives users a feeling of being there, beginning with experimental 4K broadcasting of the opening of the World Cup to be held in Brazil in 2014 and 8K broadcasting of the opening of the 2016 Rio de Janeiro Olympics and the 2020 Tokyo Olympics.

To provide attractive 4K video, it is important to design and manage the video service based on the user quality of experience (QoE). A methodology for assessing the QoE of 4K video service is therefore essential. We describe here efforts concerning QoE assessment of 4K video services.

2. 4K video QoE

The resolution of 4K video is four times that of conventional full high-definition (HD) images, so 4K video can be viewed with very high image quality. Additionally, the bit depth of 4K video is 8 bits or more, and the frame rate is two to four times higher than that of conventional full HDTV. Consequently, users can experience a higher sense of presence, sense of depth, and sense of immersiveness such as that experienced by users watching 3D movies, than with conventional HDTV [1]. We therefore need to evaluate the sense of presence, sense of depth, and sense of immersiveness as QoE factors in addition to the video quality as has been done in the past. In this article, we narrow our focus to a method of assessing 4K video quality.

3. 4K video subjective assessment methodology

Subjective quality assessment methods are the most fundamental way of assessing the QoE of video communication services. Subjective quality assessments involve visual psychological tests where participants are subjected to a video stimulus and then assess its quality based on their own subjective judgment. Special knowledge and evaluation equipment are needed to plan and implement the tests, including the selection of a video presentation method, a suitable way of measuring the assessments, and a means of adjusting the viewing conditions. Therefore, the International Telecommunication Union (ITU) has published recommendations relating to subjective quality assessments in order to regulate the viewing environments and viewing criteria with the aim of obtaining repeatable quality assessments.

Some organizations are working to develop methods to assess 4K video quality. The Video Quality Experts Group (VQEG) has established the Ultra-HD (4K) project, which has begun studies on 4K video quality assessment. This project is targeting subjective 4K video quality assessment methods and objective quality assessment methods that can estimate the results of subjective assessment of QoE from the physical characteristics of the 4K video signal or other such means. The 4K subjective quality assessment method basically follows the same process as conventional full HD subjective quality assessment methods.

Here, we introduce one of the conditions in subjective quality assessment tests that are standardized in conventional full HD subjective quality assessment methods. For example, the environment (viewing conditions) in which the video quality is assessed is set forth in ITU-R (ITU Radiocommunication sector) recommendations including BT.710 [2] and BT.1129 [3]. These recommendations state that the assessment tests must be carried out in a special room (quality assessment booth) where it is possible to create the specified environmental conditions. In addition, these recommendations define criteria for the viewing distance, luminance of the monitor, and number of participants. The quality experienced by participants (subjective quality) can vary widely, even when they are watching the same video. To reduce the variation in the assessment results, a single video sample must be scored by a large number of participants. ITU-R recommendation BT.500 [4] defines that the participants should consist of at least 15 non-experts (people who are not routinely involved in work relating to the quality of TV pictures and who have limited experience in assessment tests).

The subjective assessment method of 4K video quality follows the subjective assessment method of full HD video quality, so we discuss the possibility of applying those standards to the subjective assessments of 4K video.

3.1 Viewing distance for 4K video assessment

The viewing distance for full HD video is specified in ITU-R BT.710 to be 3H, where H is the monitor height (Fig. 1). This viewing distance is the same as the distance at which a person with a visual acuity of 1.0 on the Japanese scale cannot perceive the scanning lines on the screen [5]. The high resolution of 4K video, however, makes it possible to view from a closer distance without perceiving the scan lines. Specifically, because 4K video has a horizontal pixel count of 3840, and the viewing resolution is 60 pixels per degree of screen angle, the screen angle is about 64 degrees. For a viewing distance of 1.5H, that is about the same as the viewing angle (61°). Actually, 4K video can provide a strong sense of detail, and the image quality is maintained even when the viewer moves as close as 1.5H. We therefore assume a viewing distance of 1.5H for the subjective assessment experiments for 4K video; this distance is closer to the monitor than in subjective assessment of conventional full HD video (Fig. 1). Moreover, most 4K video quality assessment studies that have been done have used a viewing distance of 1.5H.

Fig. 1. Viewing distance.

3.2 Assessment metric of 4K video quality

The Double Stimulus Continuous Quality Scale (DSCQS) method set forth in ITU-R recommendation BT.500 is widely used in assessing the quality of systems and transmission paths used for television broadcasts. This method is particularly effective in cases where it is not possible to present the full range of quality conditions, and it is useful for simultaneously assessing the difference in quality between a reference video and an assessment video, and for assessing the absolute quality of the assessment video.

The DSCQS method, as shown in Fig. 2, involves using a pair of videos comprising the reference video and an assessment video that has been subjected to some sort of processing such as video coding. These videos are presented twice, and the assessment is performed the second time the videos are presented. The videos are presented in random order, and the participants are not told which is the reference video. The participants score the videos on a continuous quality assessment scale based on five categories, as shown in Fig. 3. The assessment scale is normalized to the range 0–100 (maximum value: 100, minimum value: 0), and the difference in video assessment values for the reference video and assessment video in each pair is calculated. These video quality differential values are averaged across all the participants to yield a DSCQS value. Because the DSCQS value is calculated from the differences in video quality, a smaller value indicates higher quality (closer to the reference video), and a larger value indicates lower quality.

Fig. 2. Flow of DSCQS method.

Fig. 3. DSCQS assessment categories.

With 4K video, the quality of the encoded video may be close to that of the reference video, depending on the encoding rate. Consequently, there is concern that non-experts will not be able to make a stable assessment of the difference in quality between a reference video and an assessment video (Fig. 4). Therefore, we conducted two subjective assessment tests using high-quality assessment videos that were close in quality to the 4K reference videos and using assessment videos that ranged in quality from low to high. The 36 participants (18 males, 18 females) ranged in age from 20 to 29 years old and had visual acuity of 1.0 on the Japanese scale. All participants joined both of the subjective assessment tests. We compared the dispersion around the average DSCQS values at the 95% confidence intervals for 4K video viewed at a distance of 1.5H with that for full HD video quality at a viewing distance of 3H (Fig. 5). The graph in Fig. 5 indicates that the dispersion in DSCQS values is about the same for 4K video and full HD video. We can thus show that non-experts can sufficiently assess the difference in quality between the reference video and 4K assessment videos that were close in quality to the reference video. It is therefore possible to use DSCQS as a metric for 4K video quality in the same way as has been done in the past.

Fig. 4. Concerns about using the DSCQS method with 4K assessment videos.

Fig. 5. Dispersion of subjective assessment values of full HD and 4K video.

4. Conclusion

We have described here a subjective technique for quality assessment of 4K video. By assessing the quality of 4K video encoded at an assumed bit-rate in 4K video services, we will be able to set the encoding rate to provide 4K video services that satisfies user expectations.

Many attractive video services are possible with 4K video services, for example, remote communication services and interactive video distribution services that allow users to select and view what they want to see. Therefore, to ensure a level of quality that satisfies users, it is important to evaluate the video quality described here as well as other sensory aspects of user experience such as the feeling of engaging in face-to-face communication, the sense of presence that seems to draw users into the proposed 4K video space, the sense of immersion, and an overall positive feeling. Techniques for evaluating such sensory experiences are particularly needed for 8K video services in the future, because viewing on larger monitors is expected.

In the future, we will develop techniques for evaluating a sense of presence and other sensory experiences that are not assessed in the conventional subjective assessment techniques, and we will continue to work on techniques to provide high quality and high sense of presence 4K and 8K video services.

References

[1]	M. Sugawara, “Advances in Super Hi-vision in 2012––Public viewings at the London Olympics, and standardization at the ITU-R,” New Breeze, Winter, pp. 4–9, 2013.
[2]	ITU-R Recommendation BT.710, “Subjective Assessment Methods for Image Quality in High-definition Television,” Nov. 1998.
[3]	ITU-R Recommendation BT.1129, “Subjective Assessment of Standard Definition Digital Television (SDTV) Systems,” Feb. 1998.
[4]	ITU-R Recommendation BT.500, “Methodology for the Subjective Assessment of the Quality of Television Pictures,” Jan. 2012.
[5]	N. Narita, M. Kanazawa, and F. Okano, “Optimum Screen Size and Viewing Distance for Viewing Ultra High-Definition and Wide-Screen Images,” J. Inst. Image Inform. TV Eng., Vol. 55, No. 5, pp. 773–780, 2001 (in Japanese).

Kimiko Kawashima: Researcher, Service Assessment Group, Communication Traffic & Service Quality Project, NTT Network Technology Laboratories.
She received the B.E. and M.E. degrees in engineering from Keio University, Kanagawa, in 2008 and 2010, respectively. She joined NTT in 2010 and has been engaged in researching the quality assessment of visual communication services. She is currently working on quality assessment of 4K video services. She is a member of the Institute of Information and Communication Engineers (IEICE).

Jun Okamoto: Senior Research Engineer, Supervisor, Service Assessment Group, Communication Traffic & Service Quality Project, NTT Network Technology Laboratories.
He received the B.S. and M.S. degrees in electrical engineering from the Science University of Tokyo in 1994 and 1996, respectively. He joined NTT in 1996 and has been engaged in the quality assessment of visual communication services. He is currently working on developing subjective and objective methods to assess audio and visual communication services. He has been contributing to VQEG (Video Quality Experts Group) since 2004. He received the Telecommunications Advancement Foundation Award in 2009 and the International Telecommunication Union Encouragement Award in Japan in 2010.

↑ TOP