Classification of methods used for assessment of text-to-speech systems according to the demands placed on the listener

Abstract A classification of different methods used for the assessment of TTS (Text-To-Speech) systems, according to the demands placed on the listener, is proposed and discussed. The classification is made according to the four traditional scale levels: the Nominal, Ordinal, Interval and Ratio level. A fifth level, the Supra-Nominal, including memory processes, is proposed. The methods are divided into qualitative, non-metric methods and quantitative, metric methods. The outcome is that the highest metric assessment level (Ratio) is not necessarily the level that places the highest demands on the listener. Quite to the contrary, the Nominal level, supporting a non-metric qualitative approach, places even higher demands on the listener. Additionally, various factors affecting the outcome regardless of at what level the assessment takes place are discussed such as the number of source and speech content conditions, dynamic response range, subjects, training, degree of user involvement and listening level, in relation to ITU-TS and ITU-R recommendations.