Comparison of two mixed methods approaches for multimodal quality evaluations: Open Profiling of Quality and Conventional Profiling

To guide the practitioner's work in choosing between assessment methods, defined criteria to compare their benefits, costs, and limitations are needed. The goal of this paper is two-fold. Firstly, we develop an extensive comparison model to guide between-method comparisons based on a literature review. The model is composed of four main criteria, called economy, excellence, implementation and assessment with a total of 24 sub-criteria. Secondly, we conduct a comparison study between two mixed methods in which a subset of criteria of the model is examined. We compare Open Profiling of Quality utilizing individuals' own vocabulary and Conventional Profiling utilizing fixed vocabulary in their descriptive evaluation. The study is conducted with naïve participants with varying 3D video qualities on a mobile device. The results compare both methods on a subset of comparison criteria and show that operationalization of the developed comparison model can provide a tool for holistic methods comparison.

[1]  C. McGarty,et al.  Research Methods and Statistics in Psychology , 2003 .

[2]  Kees Teunissen,et al.  Rapid perceptual image description (RaPID) method , 1996, Electronic Imaging.

[3]  Panos Markopoulos,et al.  How to compare usability testing methods with children participants , 2002 .

[4]  M. McTigue,et al.  Comparison of Four Sensory Evaluation Methods for Assessing Cooked Dry Bean Flavor , 1989 .

[5]  Patrick Le Callet,et al.  Tradeoffs in subjective testing methods for image and video quality assessment , 2010, Electronic Imaging.

[6]  N. Hoffart.  Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory , 2000 .

[7]  Dominik Strohmeier,et al.  Open Profiling of Quality: A Mixed Method Approach to Understanding Multimodal Quality Perception , 2010, Adv. Multim..

[8]  Dominik Strohmeier,et al.  The Extended-OPQ Method for User-Centered Quality of Experience Evaluation: A Study for Mobile 3D Video Broadcasting over DVB-H , 2011, EURASIP J. Image Video Process..

[9]  Jari Takatalo,et al.  Measuring stereoscopic image quality experience with interpretation based quality methodology , 2008, Electronic Imaging.

[10]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[11]  Miska M. Hannuksela,et al.  Acceptance Threshold: A Bidimensional Research Method for User-Oriented Quality Evaluation Studies , 2008, Int. J. Digit. Multim. Broadcast..

[12]  Satu Jumisko-Pyykkö,et al.  Descriptive quality of experience for mobile 3D video , 2010, NordiCHI.

[13]  Dominik Strohmeier,et al.  Designing for user experience: what to expect from mobile 3d tv and video? , 2008, UXTV '08.

[14]  A. Bryman.  Social Research Methods , 2001 .