No Access

Effect of Various Visual Speech Units on Language Identification Using Visual Speech Recognition

Aparna Brahme

MET’s Institute of Engineering, Adgaon, Nashik 422003, India

E-mail Address: mrsnrkale@gmail.com

Corresponding author.

Search for more papers by this author

and

Umesh Bhadade

SSBT’s College of Engineering and Technology, Bhambori, Jalgaon, India

E-mail Address: umeshbhadade@gmail.com

Search for more papers by this author

https://doi.org/10.1142/S0219467820500291Cited by:5 (Source: Crossref)

Abstract

In this paper, we describe our work in Spoken language Identification using Visual Speech Recognition (VSR) and analyze the effect of various visual speech units used to transcribe the visual speech on language recognition. We have proposed a new approach of word recognition followed by the word N-gram language model (WRWLM), which uses high-level syntactic features and the word bigram language model for language discrimination. Also, as opposed to the traditional visemic approach, we propose a holistic approach of using the signature of a whole word, referred to as a “Visual Word” as visual speech unit for transcribing visual speech. The result shows Word Recognition Rate (WRR) of 88% and Language Recognition Rate (LRR) of 94% in speaker dependent cases and 58% WRR and 77% LRR in speaker independent cases for English and Marathi digit classification task. The proposed approach is also evaluated for continuous speech input. The result shows that the Spoken Language Identification rate of 50% is possible even though the WRR using Visual Speech Recognition is below 10%, using only 1s of speech. Also, there is an improvement of about 5% in language discrimination as compared to traditional visemic approaches.

Keywords:

Remember to check out the Check out our Most Cited Articles!
Check out these titles on Image Analysis

References

1. S. Werda, W. Mahdi and A. B. Hamadou, “Lip localization and viseme classification for visual speech recognition,” International Journal of Computing & Information Sciences 5(1), 62–75 (2007). Google Scholar
2. F. Z. Chelali, K. Sadeddine and A. Djeradi, “Visual speech analysis, application to Arabic phonemes,” Special Issue of International Journal of Computer Applications (0975 – 8887) on Software Engineering, Databases and Expert Systems-SEDEXS, September 2012, pp. 29–34. Google Scholar
3. A. G. Chitu and L. J. M. Rothkrantz, “Visual speech recognition automatic system for lip reading of Dutch,” Journal of Information Technologies and Control 3, 2–9 (2009). Google Scholar
4. A. B. A. Hassanat, “Visual speech recognition. Speech and language technologies,” InTech, Available from: http://www.intechopen.com/books/speech-andlanguage-technologies/visual-speech-recognition. Google Scholar
5. A. Brahme and U. Bhadade, “Marathi digit recognition using lip geometric shape features and dynamic time warping,” in Proc. 2017 IEEE Region 10 Conference (TENCON), Malaysia, 5–8 November 2017, pp. 974–979. Google Scholar
6. A. Chitu and L. J. M. Rothkrantz, “Automatic visual speech recognition, speech enhancement, modeling and recognition-algorithms and applications,” Dr. S Ramakrishnan (Ed.), ISBN:978-953-51-0291-5, InTech, Available from: http://www.intechopen.com/books/speech-enhancement-modeling-and-recognition-algorithms-and-applications/towards-robust-visual-speech-recognition. Google Scholar
7. M. Gordan, C. Kotropoulos and I. Pitas, “A support vector machine-based dynamic network for visual speech recognition applications,” EURASIP Journal on Applied Signal Processing 11, 1248–1259 (2011). Google Scholar
8. D. Howell, S. Cox and B. Theobald, “Visual units and confusion modelling for automatic lip-reading,” Image and Vision Computing 51, 1–12 (2016), https://doi.org/10.1016/j.imavis.2016.03.003. Crossref, Web of Science, Google Scholar
9. S. Petridis, Z. Li and M. Pantic, “End-to-end visual speech recognition with LSTMS,” ICASSP 2017, arXiv:1701.05847v1[cs.CV]. Google Scholar
10. A. G. Chitu, L. J. M. Rothkrantz, J. C. Wojdel and Pascal Wiggers, “Comparison between different feature extraction techniques for audio-visual speech recognition,” Journal on Multimodal User Interfaces 1(1), 7–20, https://doi.org/10.1007/BF02884428. Crossref, Web of Science, Google Scholar
11. M. Z. Ibrahim and D. J. Mulvaney, “Geometrical-based lip-reading using template probabilistic multi-dimension dynamic time warping,” Journal of Visual Communication and Image Representation 30, 219–233 (2015). Crossref, Web of Science, Google Scholar
12. S. L. Wang, A. W. C. Liew, W. H. Lau and S. H. Leung, “An automatic lip reading system for spoken digits with limited training data,” IEEE Transactions on Circuits and Systems for Video Technology 18(12), 1760–1765 (2008). Crossref, Web of Science, Google Scholar
13. D. Yu, O. Ghita, A. Sutherland and P. F. Whelan, “A novel visual speech representation and HMM classification for visual speech recognition,” IPSJ Transactions on Computer Vision and Applications 2, 25–38 (2010). Crossref, Google Scholar
14. I. Almajai, S. Cox, R. Harvey and Y. Lan, “Improved speaker independent lip reading using speaker adaptive training and deep neural networks,” ICASSP 2016, pp. 2722–2726. Google Scholar
15. K. Kumar, T. Chen and R. M. Stern, “Profile view lip reading,” IEEE International Conference on Accoustics, Speech, and Signal Processing-ICASSP 2007, Vol. 4, pp. 429–432, https://doi.org/10.1109/ICASSP.2007.366941. Google Scholar
16. K. and S. Selvam, “Visual speech recognition using histogram of oriented displacements,” (2015). All Theses. Paper 2112. Google Scholar
17. Y. Lan, R. Harvey, B.-J. Theobald, E.-J. Ong and R. Bowden, “Comparing visual features for lip reading,” AVSP 2009, Norwich, 10–13 September 2009, pp. 102–106. Google Scholar
18. Y. M. Assael, B. Shillingford, S. Whiteson and N. de Freitas, “LIPNET: End-to-end sentence-level lipreading,” arXiv:1611.01599v2 [cs.LG] 16 Dec 2016. Google Scholar
19. J. S. Chung, A. Senior, O. Vinyals and A. Zisserman, “Lip reading sentences in the wild,” arXiv:1611.05358v2 [cs.CV] 30 Jan 2017. Google Scholar
20. B. Shillingford, Y. Assael, M. W. Hoffman, T. Paine, C. Hughes, U. Prabhu, H. Liao, H. Sak, K. Rao, L. Bennett, M. Mulville, B. Coppin, B. Laurie, A. Senior and N. de Freitas, “Large-scale visual speech recognition,” arXiv:1807.05162v3 [ cs.CV] 1 Oct 2018. Google Scholar
21. T. Afouras, J. S. Chung, A. Senior, O. Vinyals and A. Zisserman, “Deep audio-visual speech recognition,” arXiv:1809.02108v2 [cs.CV] 22 Dec 2018. Google Scholar
22. A. Waibel, P. Geutner, L. M. Tomokiyo, T. Schultz and M. Woszczyna, “Multilinguality in speech and spoken language systems,” Proc. IEEE 88(8), 1181–1190 (2000). Crossref, Web of Science, Google Scholar
23. C. Chelba, T. Hazen and M. Saraclar, “Retrieval and browsing of spoken content,” IEEE Signal Process. Mag. 25(3), 39–49 (2008). Google Scholar
24. H. Li, B. Ma and K. A. Lee, “Spoken language recognition: From fundamentals to practice,” Proceedings of the IEEE 101(5), 1136–1159 (2013). Crossref, Web of Science, Google Scholar
25. E. Ambikairajah, H. Li, L. Wang, B. Yin and V. Sethu, “Language identification: A tutorial,” IEEE Circ. Syst. Mag. 11, 82–108 (2011). Google Scholar
26. M. A. Zissman and K. M. Berkling, “Automatic language identification,” Elsevier Speech Communication 35, 115–124 (2001). Crossref, Web of Science, Google Scholar
27. M. A. Zissman, “Comparison of four approaches to automatic language identification of telephone speech,” IEEE Transactions on Speech and Audio Processing 4(1), 31–44 (1996). Crossref, Google Scholar
28. A. Amine, Z. Elberrichi and M. Simonet, “Automatic language identification: An alternative unsupervised approach using a new hybrid algorithm,” International Journal of Computer Science and Applications 7(1), 94–107 (2010). Google Scholar
29. P. A. T. Carrasquillo, D. A. Reynolds and J. R. Deller, “Language identification using Gaussian mixture model Tokenization,” IEEE Conference on ICASSP, May 2002, pp. I-757–I-760. Google Scholar
30. P. Roy and P. K. Das, “Language identification of Indian languages based on Gaussian mixture models,” International Journal of Wisdom Based Computing 1(3), 54–59 (2011). Google Scholar
31. W. C. Yau, D. K. Kumar and S. P. Arjunan, “Voiceless speech recognition using dynamic visual speech features,” VisHCI 2006, Canberra, Australia (CRPIT), Vol. 56, pp. 93–101. Google Scholar
32. R. E. Ronquest, S. V. Levi and D. B. Pisoni, “Language identification from visual-only speech signals,” Attention, Perception & Psychophysics 72(6), 1601–1613 (2010). Crossref, Web of Science, Google Scholar
33. J. L. Newman and S. J. Cox, “Automatic visual-only language identification: A preliminary study,” Conference ICASSP 2009, pp. 4345–4358, https://doi.org/10.1109/ICASSP.2009.4960591. Google Scholar
34. J. L. Newman and S. J. Cox, “Language identification using visual features,” IEEE Transactions on Audio, Speech, and Language Processing 20(7), 1936–1947 (2012). Crossref, Google Scholar
35. A. B. Hassanat, “Visual words for automatic lip-reading,” Dec2009, arXiv:1409.6689v1[cs.CV]. Google Scholar
36. A. Brahme and U. Bhadade, “Lip detection and lip geometric feature extraction using constrained local model for spoken language identification using visual speech recognition,” Indian Journal of Science and Technology 9(32), 1–7 (2016). Crossref, Google Scholar
37. S. S. Morade and S. Patnaik, “A novel lip reading algorithm by using localised ACM and HMM: Tested for digit recognition,” Optik 125, 5181–5186 (2014). Crossref, Web of Science, Google Scholar
38. A. B. A. Hassanat and S. Jassim, “Color-based lip localization method,” in Proceedings of SPIE- The International Society for Optical Engineering, April 2010, Vol. 7708. Google Scholar
39. T. F. Cootes, A. Hill, C. J. Taylor and J. Haslam, “The use of active shape models for locating structures in medical images,” Image and Vision Computing 12(6), 355–365 (1994). Crossref, Web of Science, Google Scholar
40. T. F. Cootes, C. J. Taylor, D. H. Cooper and J. Graham, “Active shape models their training and application,” Computer Vision and Image Understanding 61(1), 38–59 (1995). Crossref, Web of Science, Google Scholar
41. T. F. Cootes, G. J. Edwards and C. J. Taylor, “Active appearance models,” in Proc. European Conf. Comput Vision (Springer, June 1998), Vol. 2, pp. 484–498. Google Scholar
42. M. Kass, A. Witkin and D. Terzopoulos, “Snakes: Active contour model,” International Journal of Computer Vision 1, 321–331 (1987). Crossref, Web of Science, Google Scholar
43. N. Eveno, A. Caplier and P. Y. Coulon, “New color transformation for lips segmentation,” in Proc. IEEE 4th Workshop Multimedia Signal Proc, France (2001), pp. 3–8. Crossref, Google Scholar
44. T. Wark, S. Sridharan and V. Chandran, “An approach to statistical lip modeling for speaker identification via chromatic feature extraction,” in Proc. 4th Int. Conf. Pattern Recognition, Brisbane, Australia (1998), pp. 123–125. Google Scholar
45. X. Yan, Constrained local model for face alignment, A tutorial,” Available from: http://sites.google.com./site/xgyanhome/home/projects/CLM-implementation (accessed on 28 August 2015). Google Scholar
46. Z. Zhang, P. Luo, C. C. Loy and X. Tang, “Learning deep representation for face alignment with auxiliary attributes,” IEEE Trans. Pattern Anal. Mach. Intell. 38(5), 918–930 (2016), https://doi.org/10.1109/tpami.2015.2469286. PMID: 27046839. Crossref, Web of Science, Google Scholar
47. Z. Zhang, P. Luo, C. C. Loy and X. Tang, “Facial landmark detection by deep multi-task learning,” in Computer Vision – ECCV 2014, D. Fleet, T. Pajdla, B. Schiele and T. Tuytelaars (eds.), Lecture Notes in Computer Science, Vol. 8694, Springer, Cham, https://doi.org/10.1007/978-3-319-10599-4_7. Google Scholar
48. C. Tian, G. Sun, Q. Zhang, W. Wang and T. C. Y. Sun, “Integrating sparse and collaborative representation classifications for image classification,” International Journal of Image and Graphics 17(02), 1–13 (2017), https://doi.org/10.1142/S0219467817500073. Link, Google Scholar
49. J. Wen, X. Fang, J. Cui, L. Fei, K. Yan, Y. Chen and Y. Xu, “Robust sparse linear discriminant analysis,” IEEE Transactions on Circuits and Systems for Video Technology 29(2), 390–403 (2019), https://doi.org/10.1109/TCSVT.2018.2799214. Crossref, Web of Science, Google Scholar
50. R. Zhu and F. Dornaika, “Joint graph based embedding and feature weighting for image classification,” Pattern Recognition 93, 458–469 (2019), https://doi.org/10.1016/j.patcog.2019.05.004. Crossref, Web of Science, Google Scholar
51. W. C. Yau, D. K. Kumar and S. P. Arjunan, “Visual speech recognition using dynamic features and support vector machines,” International Journal of Image and Graphics 08(03), 419–437 (2008). Link, Google Scholar
52. C. C. Chang and C. J. Lin, “Libsvm: A library for support vector machines,” ACM Transactions on Intelligent Systems and Technology (TIST) 2(3), 1–27 (2011). Crossref, Web of Science, Google Scholar
53. L. R. Rabiner, “A tutorial on hidden Markov models and selected applications in speech recognition,” Proceedings of the IEEE 77(2), 257–286 (1989). Crossref, Web of Science, Google Scholar
54. H. Sakoe and S. Chiba, “Dynamic programming algorithm optimization for spoken word recognition,” IEEE Transactions on Acoustics, Speech and Signal Processing 26(1), 43–49 (1978). Crossref, Google Scholar
55. P. Sanguansat, “Multiple multidimensional sequence alignment using generalized dynamic time warping,” WSEAS Transactions on Mathematics 11(8), 668–678 (2012). Google Scholar
56. G. Ten Holt, M. Reinders and E. Hendriks, “Multidimensional dynamic time warping for gesture recognition,” Annual. Conference on the Advanced School for Computing and Imaging, Vol. 300 (2007). Google Scholar
57. A. G. Chitu and L. J. M. Rothkrantz, “Building a data corpus for audio-visual speech recognition,” Euromedia, April 2007, pp. 88–92. Available at http://mmi.tudelft.nl/pub/alin/APTEC-04.pdf. Google Scholar
58. A. Brahme and U. Bhadade, “Phoneme viseme mapping for Marathi language using linguistic approach,” in Proc. of Int. Conf. Global Trends in Signal Processing, Information Computing and Communication (ICGTSPICC), December 2016, pp. 152–157. Google Scholar