Design of phonetically balanced SUS test for evaluation of Slovak TTS systems

Development of speech synthesis systems requires the definition of qualitative criterions for evaluation of the output speech. We can evaluate parameters such as intelligibility or naturalness of speech at the output of the system and we can also compare the quality of various systems taking an advantage of the results obtained by these evaluations. One of the most useful evaluation methods is based on the semantically unpredictable sentences, which are being used for the subjective assessment of speech intelligibility. The paper deals with the issue of design of the test that could be used in the evaluation process of Slovak speech synthesis systems. The aim was to analyze and propose a set of semantically unpredictable sentences in Slovak language with focusing on maximizing the phonetic coverage of the proposed set of sentences. Slovak syntactic frames which are based on the most numerous morphological structures extracted from large database of Slovak language text were designed too.

[1]  A. Janicki,et al.  Taking advantage of pronunciation variation in unit selection speech synthesis for polish , 2008, 2008 3rd International Symposium on Communications, Control and Signal Processing.

[2]  Christophe d'Alessandro,et al.  A joint intelligibility evaluation of French text-to-speech synthesis systems: the EvaSy SUS/ACR campaign , 2006, LREC.

[3]  Paul Taylor,et al.  Text-to-Speech Synthesis , 2009 .

[4]  Martine Grice,et al.  The SUS test: A method for the assessment of text-to-speech synthesis intelligibility using Semantically Unpredictable Sentences , 1996, Speech Commun..

[5]  L’udov́ıt Štúr.  Slovak morphology analyzer based on Levenshtein edit operations , 2006 .

[6]  Daniel Tihelka,et al.  The Design of Czech Language Formal Listening Tests for the Evaluation of TTS Systems , 2004, LREC.

[7]  Francesc Alías,et al.  A PHONETICALLY BALANCED MODIFIED RHYME TEST FOR EVALUATING CATALAN SPEECH INTELLIGIBILITY , 2007 .

[8]  Rose Maulidiyatul Hikmah,et al.  Evaluation of text-to-speech synthesizer for Indonesian language using semantically unpredictable sentences test: IndoTTS, eSpeak, and google translate TTS , 2011, 2011 International Conference on Advanced Computer Science and Information Systems.

[9]  Ann K. Syrdal,et al.  An evaluation of the diagnostic rhyme test , 1998, Int. J. Speech Technol..

[10]  D. Hladek,et al.  Dagger: The Slovak morphological classifier , 2012, Proceedings ELMAR-2012.

[11]  Yu-Yun Chang,et al.  Evaluation of TTS Systems in Intelligibility and Comprehension Tasks , 2011, ROCLING/IJCLCLP.

[12]  Simon King,et al.  The Blizzard Challenge 2009 , 2009 .

[13]  Jian Li,et al.  Sentence level intelligibility evaluation for Mandarin text-to-speech systems using semantically unpredictable sentences , 2007, INTERSPEECH.