Declarative Evaluation of an MT system: Practical Experiences

The authors recently had the opportunity to evaluate the performance of a small commercial MT system – Globalink Translation System (GTS) – which runs on PC-type machines. A review which we published in the popular UK small systems journal Personal Computer World (1) was very much directed towards the needs of potential users. The present paper is intended as a rather fuller account of some of the difficulties encountered in trying to construct an appropriate evaluation method within a realistic time-scale. The moral of the paper is plain enough: evaluating an MT system from a userperspective is a much trickier business than most Computational Linguists might suppose.