To view PDF files

You need Adobe Reader 7.0 or later in order to read PDF files on this site.
If Adobe Reader is not installed on your computer, click the button below and go to the download site.

Letters

Objective Quality Evaluation Model for Videophone Services

Kazuhisa Yamagishi † , Toshiko Tominaga, Takanori Hayashi, and Akira Takahashi

Abstract

This article describes an objective quality evaluation model that can estimate videophone quality using quality parameters instead of captured media signals and IP (Internet protocol) packets. It enables effective design and management of videophone services. Its video quality estimation and multimedia quality integration functions were standardized as ITU-T Rec. G.1070 in 2007.

PDF
NTT Service Integration Laboratories
Musashino-shi, 180-8585 Japan
Email: yamagishi.kazuhisa@lab.ntt.co.jp

1. Introduction

Videophone services over IP (Internet protocol) will become key services in the next generation network (NGN). To provide a high-quality service for users, it is extremely important to design and manage the quality of experience (QoE) appropriately. To do this, it is desirable to develop an objective quality evaluation model that can estimate subjective quality from physical quality parameters of videophone services.

2. Objective quality measurement

Objective quality assessment [1] can be categorized into media-layer objective models, packet-layer objective models, parametric models, and hybrid models from the viewpoint of the input information. To estimate the quality perceived by users, media-layer objective models use media signals [2]–[5], packet-layer objective models use information about IP packets [6]–[9], parametric models use quality parameters [10]–[18], and hybrid models use a combination of media signals, IP packets, and quality parameters. Media-layer objective models are highly correlated with subjective quality and used for benchmarking and management. However, this approach is inconvenient for QoE planning because relationships among media quality and quality parameters are not directly considered. Packet-layer objective models are mainly used for in-service quality management [6]. Parametric models are convenient for QoE planning because they formulate the relationships among subjective quality and quality parameters and can estimate quality using quality parameters instead of using captured media signals and IP packets. They enable QoE planners to help ensure that users will be satisfied with end-to-end transmission performance, which avoids over-engineering. These models incorporate network, application, and terminal equipment parameters of high importance to QoE planners. In this article, we describe a parametric model for estimating videophone quality that can be used for application and/or network planning and monitoring.

3. Framework of the model

The framework of the parametric model is shown in Fig. 1. Its input parameters are video and speech quality parameters that are considered important in QoE planning and monitoring. The model consists of three functions for speech quality estimation, video quality estimation, and multimedia quality integration. The degradation caused by pure delay is considered only in the multimedia quality integration function.


Fig. 1. Framework of parametric model for videophone services.

1) Speech quality estimation function

This function estimates the listening speech quality using speech quality parameters. The E-model, which has been standardized as ITU-T Recommendation G.107 [11], is widely used for speech services including IP telephony. It can estimate the overall communication quality using a combination of quality factors. However, NTT proposed a new parametric model [13], [14] that can achieve better performance than the E-model and is also applicable to wideband IP telephony services. We use this NTT model for the speech quality estimation model.

2) Video quality estimation function

This function estimates viewing video quality using video quality parameters. It has three features: One is to estimate the video quality affected by coding distortion. Specifically, we consider the optimal frame rate that maximizes the video quality at each bit rate. Another feature is to create a packet loss robustness factor that indicates the degree of video quality degradation due to packet loss rate. The third is to change the coefficient tables for estimating the video quality of various implementations of videophone applications because the video quality cannot be estimated based simply on codec information. For the E-model, methodologies for deriving speech quality factors affected by the codec are provided in ITU-T Recommendation G.113, Appendix I.

3) Multimedia quality integration function

This function can be used for estimating the overall quality from the listening speech quality, viewing video quality, and end-to-end delay. It considers the individual media qualities, delay (i.e., absolute audiovisual delay and audiovisual media synchronization), and their interactions. The output of this model is multimedia quality, as shown in Fig. 1.

4. Accuracy of parametric model for videophone services

The accuracy of the speech quality estimation functions is described in [13], [14], so this section describes the accuracy of the video quality estimation and multimedia quality integration functions. Using the optimized video quality estimation function for a codec, we estimated the subjective video qualities, as shown in Fig. 2. The mean opinion score (MOS) was given by a five-grade absolute category rating (ACR) method (excellent, good, fair, poor, or bad) [19]. As shown in Fig. 2, its estimation correlates very well with subjective quality. By changing the table of coefficients calculated in advance for each codec, we found that our model could estimate video quality [17], [18]. Next, we estimated the overall quality using estimated individual media qualities and delays. The results for the multimedia quality integration function are shown in Fig. 3. The accuracy of the function was sufficient for practical use. These functions provide reliable information about what users actually require for videophone services, so this method can be used for effective design, implementation, and management of both interactive audiovisual applications and communication networks.


Fig. 2. Accuracy of video quality estimation model.


Fig. 3. Accuracy of multimedia quality integration model.

5. Quality planning using our model

Application and/or network planning to improve video quality is extremely important to avoid over-engineering and to provide users with services appropriately, as shown in Fig. 4. In this section, we show an application and network planning example. It considers the following questions.

– What is the optimal frame rate Ofr for each bit rate condition?

– What is the minimum bit rate BrMin to achieve the QoE requirement (e.g., MOS3.5)?

– How much packet loss rate PplMax is acceptable for maintaining that QoE?


Fig. 4. Example of application scenario of QoE planning and monitoring.

When the bit rate was set to 40, 82, 160, and 384 kbps, our model gave Ofr of 15, 20, 30, and 30 frames per second (fps), respectively, as shown in Fig. 5. When a QoE planner requires MOS3.5, our model enables us to find BrMin=82 kbit/s, as shown in Fig. 5. To achieve the above QoE requirement when the bit rate is set to 160 or 384 kbit/s, our model indicates that the maximum packet loss rate PplMax is about 1.2%, as shown in Fig. 6. When the QoE planner requires MOS2.5 (MOS=2.5 is the threshold that is acceptable to 50% of users), the PplMax values were about 6 and 4%, respectively. That reveals that a higher bit rate does not necessarily lead to a high-quality service. Our model is a powerful tool for reflecting such characteristics in network planning.


Fig. 5. Relationship between frame rate and video quality for each bit rate.


Fig. 6. Relationship between packet loss rate and video quality for each bit rate.

6. Conclusion

Our model based on quality parameters is a promising way of designing and monitoring the quality of videophone services. Experimental results showed that its estimates correlate very well with subjective quality, which represents the user’s perception of a service. This model will make it easy to design and manage the QoE appropriately, which is important when providing a high-quality service for users in the NGN.

References

[1] A. Takahashi, H. Yoshino, and N. Kitawaki, "Perceptual QoS Assessment Technologies for VoIP, IEEE Commun. Mag., Vol. 42, No. 7, pp. 28&ndash34, July 2004.
[2] "Perceptual Evaluation of Speech Quality (PESQ): an objective method for end-to-end speech quality assessment of narrowband telephone networks and speech codecs, ITU-T Recommendation P.862, Feb. 2001.
[3] VQEG. Video Quality Experts Group website, http://www.vqeg.org/
[4] J. Okamoto, T. Hayashi, A. Takahashi, and T. Kurita, "Verification of objective quality assessment method for arbitrary video sequence, IMQA, Sept. 2005.
[5] "Requirements for an objective perceptual multimedia quality model, ITU-T Recommendation J.148, 2003.
[6] T. Hayashi, M. Masuda, T. Tominaga, and K. Yamagishi, "Non-intrusive QoS Monitoring Method for Realtime Telecommunication Services, NTT Technical Review, Vol. 4, No. 4, pp. 35&ndash59, 2006.
[7] A. Clark, "Modeling the Effects of Burst Packet Loss and Recency on Subjective Voice Quality, IP Telephony Workshop, 2001.
[8] "Conformance testing for narrowband voice over IP transmission quality assessment models, ITU-T Recommendation P.564, July 2006.
[9] M. Masuda and T. Hayashi, "Non-intrusive Quality Monitoring Method of VoIP Speech Based on Network Performance Metrics, IEICE Trans. Commun., Vol. E89-B, No. 2, pp. 304&ndash312, Feb. 2006.
[10] A. Takahashi, A. Kurashima, and H. Yoshino, "Objective Assessment Methodology for Estimating Conversational Quality in VoIP, IEEE Trans. on Speech Audio and Language Processing, Vol. 14, No. 6, 1983&ndash1993, Nov. 2006.
[11] "The E-model, a computational model for use in transmission planning, ITU-T Recommendation G.107, Mar. 2005.
[12] A. Takahashi, "Opinion model for estimating conversational quality of VoIP, Proc. of IEEE ICASSP 2004, Vol. III, pp. 1072&ndash1075, May 2004.
[13] A. Takahashi, A. Kurashima, and H. Yoshino, "Objective Assessment Methodology for Estimating Conversational Quality in VoIP, IEEE Trans. on Speech Audio and Language Processing, Vol. 14, No. 6, 1983&ndash1993, Nov. 2006.
[14] A. Takahashi, A. Kurashima, and H. Aoki, "Quality assessment of wideband speech communication services, NTT Technical Review, Vol. 4, No. 4, pp. 47&ndash51, 2006.
[15] K. Yamagishi and T. Hayashi, "Analysis of psychological factors for quality assessment of interactive multimodal service, Electronic Imaging 2005, pp. 130&ndash138, Jan. 2005.
[16] K. Yamagishi and T. Hayashi, "Opinion Model using Psychological Factors for Interactive Multimodal Services, IEICE Trans. Commun., Vol. E89-B, No. 2, pp. 281&ndash288, Feb. 2006.
[17] K. Yamagishi and T. Hayashi, "Verification of Video Quality Opinion Model for Videophone Services, 2nd ISCA/DEGA Tutorial & Research Workshop on Perceptual Quality of Systems, pp. 143&ndash148, Sept. 2006.
[18] K. Yamagishi and T. Hayashi, "Opinion Model for Estimating Video Quality of Videophone Services, IEEE GLOBECOM 2006, QRP-1, Nov. 2006.
[19] "Subjective video quality assessment methods for multimedia applications, ITU-T Recommendation P.910, Sept. 1999.
Kazuhisa Yamagishi
Research Engineer, Communication Traffic & Service Quality Project, NTT Service Integration Laboratories.
He received the B.E degree in electrical engineering from Tokyo University of Science, Chiba, in 2001 and the M.E. degree in electronics, information, and communication engineering from Waseda University, Tokyo, in 2003. Since joining NTT Service Integration Laboratories in 2003, he has been engaged in subjective quality assessment of multimedia telecommunications and image coding. Currently, he is working on the quality assessment of interactive multimodal services over IP networks. He has been contributing to ITU-T SG12 since 2006. He is a member of the Institute of Electronics, Information and Communication Engineers (IEICE) of Japan and received the Young Investigators°« Award from IEICE in 2007.
Toshiko Tominaga
Research Engineer, Communication Traffic & Service Quality Project, NTT Service Integration Laboratories.
She received the B.E. and M.E. degrees in industrial engineering from the University of Electro-Communications, Tokyo, in 1987 and 1989, respectively. Since joining NTT Laboratories in 1989, she has been studying quality management methods for telecommunication services. She is currently working on quality management methods for video communications over IP networks. She is a member of IEICE.
Takanori Hayashi
Senior Research Engineer, Communication Traffic & Service Quality Project, NTT Service Integration Laboratories.
He received the B.E., M.E., and Ph.D degrees in engineering from the University of Tsukuba, Ibaraki, in 1988, 1990, and 2007, respectively. Since joining NTT Laboratories in 1990, he has been engaged in the subjective quality assessment of multimedia telecommunications and network performance measurement methods. He is currently working on a multimodal quality assessment method over IP networks. He is a member of IEICE.
Akira Takahashi
Group Manager, Service Assessment Group, Communication Traffic & Service Quality Project, NTT Service Integration Laboratories.
He received the B.S. degree in mathematics from Hokkaido University, Hokkaido, in 1988, the M.S. degree in electrical engineering from the California Institute of Technology, Pasadena, USA, in 1993, and the Ph.D. degree in engineering from the University of Tsukuba, Ibaraki, in 2007. He joined NTT Laboratories in 1988 and has been engaged in research into the quality assessment of speech and audio telecommunications. Currently, he is engaged in managing the research on multimedia quality assessment. He has been contributing to ITU-T SG12 since 1994 and is currently a Rapporteur of Question 13/12. He received the Telecommunication Technology Committee Award in Japan in 2004 and the ITU-AJ Award in Japan in 2005. He also received the Best Tutorial Paper Award of IEICE in Japan in 2006. He is a member of IEEE and IEICE.

↑ TOP