Research ArticlesNo Access

Efficient Embedded Decoding of Neural Network Language Models in a Machine Translation System

Francisco Zamora-Martinez

R&D Department, das-Nano S. L., Polígono Industrial Talluntxe II, Tajonar 31192, Spain

E-mail Address: pakozm@gmail.com

Search for more papers by this author

and

Maria Jose Castro-Bleda

Departamento de Sistemas Informáticos y Computación, Universitat Politècnica de València, València, Spain

E-mail Address: mcastro@dsic.upv.es

Corresponding author.

Search for more papers by this author

https://doi.org/10.1142/S0129065718500077Cited by:9 (Source: Crossref)

Abstract

Neural Network Language Models (NNLMs) are a successful approach to Natural Language Processing tasks, such as Machine Translation. We introduce in this work a Statistical Machine Translation (SMT) system which fully integrates NNLMs in the decoding stage, breaking the traditional approach based on $n$ -best list rescoring. The neural net models (both language models (LMs) and translation models) are fully coupled in the decoding stage, allowing to more strongly influence the translation quality. Computational issues were solved by using a novel idea based on memorization and smoothing of the softmax constants to avoid their computation, which introduces a trade-off between LM quality and computational cost. These ideas were studied in a machine translation task with different combinations of neural networks used both as translation models and as target LMs, comparing phrase-based and $N$ -gram-based systems, showing that the integrated approach seems more promising for $N$ -gram-based systems, even with nonfull-quality NNLMs.

Keywords:

References

1. H. Schwenk, Continuous space language models, Comput. Speech Lang. 21(3) (2007) 492–518. Crossref, Web of Science, Google Scholar
2. D. Bahdanau, K. Cho and Y. Bengio, Neural machine translation by jointly learning to align and translate, arXiv:1409.0473v6. Google Scholar
3. J. Devlin, R. Zbib, Z. Huang, T. Lamar, R. Schwartz and J. Makhoul, Fast and robust neural network joint models for statistical machine translation, in Proc. 52nd Annual Meeting of the Association for Computational Linguistics (ACL) (Baltimore, Maryland, USA, 2014), pp. 1370–1380. Google Scholar
4. Y. LeCun, Y. Bengio and G. Hinton, Deep learning, Nature 521 (2015) 436–444. Crossref, Medline, Web of Science, Google Scholar
5. A. Ortiz, J. Munilla, J. M. Górriz and J. Ramírez, Ensembles of deep learning architectures for the early diagnosis of the Alzheimers disease, Int. J. Neural Syst. 26(7) (2016) 1650025. Link, Web of Science, Google Scholar
6. K. Papineni, S. Roukos and T. Ward, Maximum likelihood and discriminative training of direct translation models, in Proc. Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP) (IEEE, Seattle, WA, USA, 1998), pp. 189–192. Google Scholar
7. F. Och and H. Ney, Discriminative training and maximum entropy models for statistical machine translation, in Proc. 40th Annual Meeting of the Association for Computational Linguistics (ACL) (Philadelphia, Pennsylvania, USA, 2002), pp. 295–302. Google Scholar
8. H. Schwenk and J.-L. Gauvain, Connectionist language modeling for large vocabulary continuous speech recognition, in Proc. Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP) (IEEE, Orlando, Florida, USA, 2002), pp. 765–768. Google Scholar
9. Y. Bengio, R. Ducharme, P. Vincent and C. Jauvin, A neural probabilistic language model, J. Mach. Learn. Res. 3(2) (2003) 1137–1155. Google Scholar
10. M. J. Castro-Bleda and F. Prat, New directions in connectionist language modeling, in Computational Methods in Neural Modeling, LNCS (Springer-Verlag, Berlin, 2003), pp. 598–605. Crossref, Google Scholar
11. H. Schwenk, D. Dèchelotte and J. L. Gauvain, Continuous space language models for statistical machine translation, in Proc. Joint Conf. ACL/Coling (Sydney, Australia, 2006), pp. 723–730. Google Scholar
12. Y. Bengio, Neural net language models, Scholarpedia 3(1) (2008) 3881. Crossref, Google Scholar
13. L. H. Son, A. Allauzen and F. Yvon, Continuous space translation models with neural networks, in Proc. Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT) (Montreal, Canada, 2012), pp. 39–48. Google Scholar
14. A. Vaswani, Y. Zhao, V. Fossum and D. Chiang, Decoding with large-scale neural language models improves translation, in Proc. Empirical Methods in Natural Language Processing (EMNLP) (Seattle, USA, 2013), pp. 1387–1392. Google Scholar
15. F. Morin and Y. Bengio, Hierarchical probabilistic neural network language model, in Proc. 10th Int. Workshop on Artificial Intelligence and Statistics (AISTATS) (The Savannah Hotel, Barbados, 2005), pp. 246–252. Google Scholar
16. A. Mnih and G. Hinton, A scalable hierarchical distributed language model, Adv. Neural Inf. Process. Syst. 21 (2009) 1081–1088. Google Scholar
17. T. Mikolov, A. Deoras, D. Povey, L. Burget and J. Cernocký, Strategies for training large-scale neural network language models, in Proc. Workshop on Automatic Speech Recognition and Understanding (ASRU) (IEEE, Waikoloa, HI, USA, 2011), pp. 196–201. Google Scholar
18. L. Hai-Son, I. Oparin, A. Alluzen, J.-L. Gauvain and F. Yvon, Structured output layer neural network language model, in Proc. Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP) (IEEE, Prague, Czech Republic, 2011), pp. 5524–5527. Google Scholar
19. H. S. Le, I. Oparin, A. Messaoudi, A. Allauzen, J.-L. Gauvain and F. Yvon, Large vocabulary soul neural network language model, in Proc. Interspeech (Florence, Italy, 2011), pp. 1469–1472. Google Scholar
20. Y. Shi, W.-Q. Zhang, M. Cai and J. Liu, Efficient one-pass decoding with nnlm for speech recognition, IEEE Signal Process. Lett. 21 (2014) 377–381. Crossref, Web of Science, Google Scholar
21. M. Gutmann and A. Hyvärinen, Noise-contrastive estimation: A new estimation principle for unnormalized statistical models, in Proc. Int. Conf. Artificial Intelligence and Statistics (AISTATS) (Sardinia, Italy, 2010), pp. 297–304. Google Scholar
22. A. Mnih and Y. W. Teh, A fast and simple algorithm for training neural probabilistic language models, in Proc. 29th Int. Conf. Machine Learning (ICML) (Edinburgh, Scotland, 2012), pp. 1751–1758. Google Scholar
23. V. Mnih and G. E. Hinton, Learning to label aerial images from noisy data, in Proc. 29th Int. Conf. Machine Learning (ICML) (Edinburgh, Scotland, 2012), pp. 567–574. Google Scholar
24. A. Mnih and K. Kavukcuoglu, Learning word embeddings efficiently with noise-contrastive estimation, Adv. Neural Inf. Process. Syst. 26 (2013) 2265–2273. Google Scholar
25. T. Mikolov, S. Kombrink, L. Burget, J. Cernocký and S. Khudanpur, Extensions of recurrent neural network language model, in Proc. Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP) (IEEE, Prague, Czech Republic, 2011), pp. 5528–5531. Google Scholar
26. Y. Hu, M. Auli, Q. Gao and J. Gao, Minimum translation modeling with recurrent neural networks, in Proc. 14th Conf. European Chapter of the Association for Computational Linguistics (EACL) (Gothenburg, Sweden, 2014), pp. 20–29. Google Scholar
27. A. Graves, S. Fernández, F. Gomez and J. Schmidhuber, Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks, in Proc. 23rd Int. Conf. Machine learning (ICML) (Pittsburgh, Pennsylvania, USA, 2006), pp. 369–376. Google Scholar
28. A. Emami and L. Mangu, Empirical study of neural network language models for arabic speech recognition, in Proc. Workshop on Automatic Speech Recognition Understanding (ASRU) (IEEE, Kyoto, Japan, 2007), pp. 147–152. Google Scholar
29. F. Zamora-Martínez, V. Frinken, S. España-Boquera, M. J. Castro-Bleda, A. Fischer and H. Bunke, Neural network language models for offline handwriting recognition, Pattern Recognit. 47(4) (2014) 1642–1652. Crossref, Web of Science, Google Scholar
30. J. Park, X. Liu, M. J. Gales and P. C. Woodland, Improved neural network based language modelling and adaptation, in Proc. Interspeech (Makuhari, Japan, 2010), pp. 26–30. Google Scholar
31. H. Schwenk and P. Koehn, Large and diverse language models for statistical machine translation, in Proc. Int. Joint Conf. Natural Language Processing (IJCNLP) (Hyderabad, India, 2008), pp. 661–666. Google Scholar
32. F. Zamora-Martínez, M. J. Castro-Bleda and S. España-Boquera, Fast evaluation of connectionist language models, in Proc. Int. Work Conf. Artificial Neural Networks (IWANN) (Springer, Salamanca, Spain, 2009), pp. 33–40. Google Scholar
33. C. Allauzen, M. Mohri and B. Roark, Generalized algorithms for constructing statistical language models, in Proc. 41st Annual Meeting on Association for Computational Linguistics (ACL) (Sapporo, Japan, 2003), pp. 40–47. Google Scholar
34. E. Arisoy, S. F. Chen, B. Ramabhadran and A. Sethy, Converting neural network language models into back-off language models for efficient decoding in automatic speech recognition, in Proc. Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP) (IEEE, Vancouver, BC, Canada, 2013), pp. 8242–8246. Google Scholar
35. R. Wang, M. Utiyama, I. Goto, E. Sumita, H. Zhao and B.-L. Lu, Converting continuous-space language models into N-gram language models for statistical machine translation, in Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP) (Seattle, Washington, USA, 2013), pp. 845–850. Google Scholar
36. P. Koehn, F. J. Och and D. Marcu, Statistical phrase-based translation, in Proc. 2003 Conf. North American Chapter of the Association for Computational Linguistics on Human Language Technologies (NAACL HLT) (Edmonton, Canada, 2003), pp. 48–54. Google Scholar
37. P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin and E. Herbst, Moses: Open-source toolkit for statistical machine translation, in Proc. Association for Computational Linguistics (ACL) (Prague, Czech Republic, 2007), pp. 177–180. Google Scholar
38. F. Prat, F. Casacuberta and M. J. Castro, Machine translation with grammar association: Combining neural networks and finite-state models, in Proc. Second Workshop on Natural Language Processing and Neural Networks (Tokyo, Japan, 2001), pp. 53–60. Google Scholar
39. F. Casacuberta, E. Vidal and J. M. Vilar, Architectures for speech-to-speech translation using finite-state models, in Proc. Workshop on Speech-to-Speech Translation: Algorithms and Systems (Stroudsburg, PA, USA, 2002), pp. 39–44. Google Scholar
40. F. Casacuberta and E. Vidal, Machine translation with inferred stochastic finite-state transducers, Comput. Linguist. 30 (2004) 205–225. Crossref, Web of Science, Google Scholar
41. J. B. Mariño, R. E. Banchs, J. M. Crego, A. de Gispert, P. Lambert, J. A. Fonollosa and M. R. Costa-jussà, N-gram-based machine translation, Comput. Linguist. 32 (2006) 527–549. Crossref, Web of Science, Google Scholar
42. M. R. Costa-jussà, J. M. Crego, P. Lambert, M. Khalilov, J. A. Fonollosa, J. B. Mariño and R. E. Banchs, Ngram-based statistical machine translation enhanced with multiple weighted reordering hypotheses, in Proc. Second Workshop on Statistical Machine Translation (WMT) (Prague, Czech Republic, 2007), pp. 167–170. Google Scholar
43. H.-S. Le, T. Lavergne, A. Allauzen, M. Apidianaki, L. Gong, A. Max, A. Sokolov, G. Wisniewski and F. Yvon, Limsi@wmt12, in Proc. Workshop on Statistical Machine Translation (WMT) (Montreal, Quebec, Canada, 2012), pp. 330–337. Google Scholar
44. P. F. Brown, V. J. Della-Pietra, S. A. Della-Pietra and R. L. Mercer, The mathematics of statistical machine translation: Parameter estimation, Comput. Linguist. 19(2) (1993) 263–311. Google Scholar
45. S. España-Boquera, F. Zamora-Martínez, M. J. Castro-Bleda and J. Gorbe-Moya, Efficient BP algorithms for general feedforward neural networks, in Bio-Inspired Modeling of Cognitive Tasks, LNCS (Springer, Berlin, 2007), pp. 327–336. Crossref, Google Scholar
46. F. Zamora-Martínez, S. España-Boquera, J. Gorbe-Moya, J. Pastor-Pellicer and A. Palacios-Corella, APRIL-ANN toolkit, A pattern recognizer in lua with artificial neural networks (2013), https://github.com/pakozm/april-ann. Google Scholar
47. F. Zamora-Martínez, M. J. Castro-Bleda and H. Schwenk, N-gram-based machine translation enhanced with neural networks for the French–English BTEC-IWSLT'10 task, in Proc. 7th Int. Workshop on Spoken Language Translation (IWSLT) (Paris, France, 2010), pp. 45–52. Google Scholar
48. F. J. Och and H. Ney, A systematic comparison of various statistical alignment models, Comput. Linguist. 29(1) (2003) 19–51. Crossref, Web of Science, Google Scholar
49. A. Stolcke, SRILM: An extensible language modeling toolkit, in Proc. Int. Conf. Spoken Language Processing (ICSLP) (Denver, Colorado, USA, 2002), pp. 901–904. Google Scholar
50. T. Mikolov, S. Kombrink, L. Burget, J. Cernocky and S. Khudanpur, Extensions of recurrent neural network language model, in Proc. Int. Conf. Acoustics, Speech and Signal Processing (ICASSP) (IEEE, Prague, Czech Republic, 2011), pp. 5528–5531. Google Scholar
51. K. Papineni, S. Roukos, T. Ward and W.-J. Zhu, BLEU: A method for automatic evaluation of machine translation, in Proc. 40th Annual Meeting on Association for Computational Linguistics (ACL) (Stroudsburg, PA, USA, 2002), pp. 311–318. Google Scholar
52. M. Snover, B. Dorr, R. Schwartz, L. Micciulla and J. Makhoul, A study of translation edit rate with targeted human annotation, in Proc. Association for Machine Translation in the Americas (Cambridge, Massachusetts, USA, 2006), pp. 223–231. Google Scholar
53. P. Koehn, Statistical significance tests for machine translation evaluation, in Proc. Empirical Methods in Natural Language Processing (EMNLP) (Barcelona, Spain, 2004), pp. 388–395. Google Scholar