Methods of Quality Assessment for Large Sample Sets

Psychophysical evaluation of large sample sets was studied with reference to the International Newspaper Colour Quality Club Jury Evaluation, in which 150 prints of the same image are assessed by category judgement under 10 different quality attributes. In a series of experiments using sub-sets of the CQC prints, some psychophysical techniques which could affect the reliability and precision of the results were evaluated. It was also possible to gain insights into the relationship between the different psychophysical methods. On the basis of these results a number of modifications to the category judgement task used in the Jury Evaluation are proposed. These include the adoption of an anchor image whose scores on the quality attributes are defined in a preliminary observer task; and a reduction in the number of attributes and judgement categories. The paper addresses a major problem in using psychophysics for objective visual assessment of samples, which is the difficulty in efficiently scaling up the number of samples in an experiment. Many studies have relatively limited numbers of samples or inadequate experimental design or data analysis methods, and this paper shows that anchored category judgement is an appropriate technique for larger sample sets. The results provide a rigorous basis for large-scale quality evaluations carried out in industry and in research labs. The work applies to sample sets in which a single image is reproduced by many different methods or different printers. In a continuation of this work I am investigating methods of assessing reproduction quality where the number of different images is large. When complete, this will enable researchers to ensure that quality assessments are not image-dependent. Green coordinated and performed psychophysical experiments and data analysis. S. Lindberg contributed statistical development of the work.