Consistency in the Evaluation Methods of Machine Translation Quality

Machine translation research contains lots of howlers and misconceptions but the classics of all time and my personal favorite is as follows: Once an MT program translated “The spirit is willing but the flesh is weak” into Russian and the back translation was “The vodka is good but the meat is rotten”. This study is written to understand how far the quality has improved. It is expected that the study will contribute both to human translation and machine translation research by providing a comprehensive evaluation by human annotators and by providing an error categorization on the outputs of machine translation programs from English into Turkish.