On Designing Paired Comparison Experiments for Subjective Multimedia Quality Assessment
- 26 November 2013
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 16 (2), 564-571
- https://doi.org/10.1109/tmm.2013.2292590
Abstract
This paper investigates the issue of designing paired comparison-based subjective quality assessment experiments for reliable results. In particular, the convergence behavior of the quality scores estimated from paired comparison results is considered. Via an extensive computer simulation experiment, the estimation performance in terms of the root mean squared error, the rank order correlation coefficient, and the change of the estimated scores with respect to the number of subjects are mathematically modeled. Furthermore, it is confirmed that the models coincide with the theoretical convergence behavior. Issues such as the effect of human errors and the underlying distribution of the true quality scores are also examined.Keywords
This publication has 21 references indexed in Scilit:
- Analysis and improvement of a paired comparison method in the application of 3DTV subjective experimentPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2012
- Models for Paired Comparison Data: A Review with Emphasis on Dependent DataStatistical Science, 2012
- Paired comparison-based subjective quality assessment of stereoscopic imagesMultimedia Tools and Applications, 2012
- A comparative study of image retargetingACM Transactions on Graphics, 2010
- Selecting numerical scales for pairwise comparisonsReliability Engineering & System Safety, 2010
- Empirical formula for creating error bars for the method of paired comparisonJournal of Electronic Imaging, 2006
- Paired comparisons in visual perception studies using small sample sizesDisplays, 2005
- Asymptotics when the number of parameters tends to infinity in the Bradley-Terry model for paired comparisonsThe Annals of Statistics, 1999
- Rank Analysis of Incomplete Block Designs: I. The Method of Paired ComparisonsBiometrika, 1952
- On the Method of Paired ComparisonsBiometrika, 1940