On Evaluation Metrics for Output Stability of Machine Translation System
- 1 January 2017
- journal article
- Published by Japanese Society for Artificial Intelligence in Transactions of the Japanese Society for Artificial Intelligence
- Vol. 32 (5)
- https://doi.org/10.1527/tjsai.d-h33
Abstract
This paper presents a novel metric for evaluating stability of machine translation system. A stable system indicates that it keeps almost the same outputs given the inputs with slight changes. In this paper, we propose a stability metric by exploiting TER metric for evaluating the differences between the two texts. We have built an evaluation data set, and demonstrate that a neural-based method is unstable rather than a statistical-based method, while the former outperforms the latter.Keywords
This publication has 2 references indexed in Scilit:
- Effective Approaches to Attention-based Neural Machine TranslationPublished by Association for Computational Linguistics (ACL) ,2015
- BLEUPublished by Association for Computational Linguistics (ACL) ,2001