No Access

Evaluating the Pertinence of Pose Estimation model for Sign Language Translation

K. Amrutha

Department of Computer Science, CHRIST (Deemed to be University), Bengaluru, Karnataka, India

Search for more papers by this author

and

P. Prabu

Department of Computer Science, CHRIST (Deemed to be University), Bengaluru, Karnataka, India

E-mail Address: prabu.p@christuniversity.in

Corresponding author.

Search for more papers by this author

https://doi.org/10.1142/S1469026823410092Cited by:18 (Source: Crossref)

This article is part of the issue:

Special Issue on Smart Fuzzy Optimization for Decision-Making in Uncertain Environments
Guest Editors: Er Meng Joo, Danilo Pelusi and Shiping Wen

Abstract

Sign Language is the natural language used by a community that is hearing impaired. It is necessary to convert this language to a commonly understandable form as it is used by a comparatively small part of society. The automatic Sign Language interpreters can convert the signs into text or audio by interpreting the hand movements and the corresponding facial expression. These two modalities work in tandem to give complete meaning to each word. In verbal communication, emotions can be conveyed by changing the tone and pitch of the voice, but in sign language, emotions are expressed using nonmanual movements that include body posture and facial muscle movements. Each such subtle moment should be considered as a feature and extracted using different models. This paper proposes three different models that can be used for varying levels of sign language. The first test was carried out using the Convex Hull-based Sign Language Recognition (SLR) finger spelling sign language, next using a Convolution Neural Network-based Sign Language Recognition (CNN-SLR) for fingerspelling sign language, and finally pose-based SLR for word-level sign language. The experiments show that the pose-based SLR model that captures features using landmark or key points has better SLR accuracy than Convex Hull and CNN-based SLR models.

Keywords:

Remember to check out the Most Cited Articles!
Check out these titles in artificial intelligence!

References

1. R. Feris et al., Exploiting depth discontinuities for vision-based fingerspelling recognition, in IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Washington, DC. Google Scholar
2. K. Grobel and M. Assan, Isolated sign language recognition using Hidden Markov models, in Proc. IEEE Int. Conf. Systems, Man, and Cybernetics, Vol. 1, pp. 162–167, Orlando, FL, USA (1997). Crossref, Google Scholar
3. H. D. Yang, S. Sclaro and S. W. Lee, Sign language spotting with a threshold model based on conditional random fields, IEEE Trans. Pattern Anal. Mach. Intell. 31(7) (2009) 1264–1277, https://doi.org/10.1109/TPAMI.2008.172. Crossref, Google Scholar
4. A. Wadhawan and P. Kumar, Deep learning-based sign language recognition system for static signs, Neural Comput. Appl. 32(12) (2020) 7957–7968, https://doi.org/10.1007/s00521-019-04691-y. Crossref, Google Scholar
5. D. L. Jaffe, W. S. Harwin and D. E. Wilmington, The development of a third-generation fingerspelling hand, TITLE Engineering the ADA from Vision to Reality with Technology (16th, Las Vegas, Nevada, June 12–17, 1993), p. 175. Google Scholar
6. S. Sidney Fels and G. E. Hinton, Glove-talk: A neural network interface between a data-glove and a speech synthesizer, IEEE Trans. Neural Networks 4(1) (1993) 2–8, https://doi.org/10.1109/72.182690. Crossref, Google Scholar
7. G. Marin, F. Dominio and P. Zanuttigh, Hand gesture recognition with leap motion and kinect devices, in 2014 IEEE Int. Conf. Image Processing (2014), pp. 1565–1569. Crossref, Google Scholar
8. P. Kumar, R. Saini, S. Kumar Behera, D. P. Dogra and P. P. Roy, Real-time recognition of sign language gestures and air-writing using leap motion, in 2017 15th IAPR Int. Conf. Machine Vision Applications (MVA), Nagoya, Japan, 2017, pp. 157–160. Crossref, Google Scholar
9. L. Lamberti and F. Camastra, Real-time hand gesture recognition using a color glove, in Lecture Notes in Computer Science, Subseries in Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics, Vol. 6978 (2011), pp. 365–373. https://doi.org/10.1007/978-3-642-24085-0_38 Google Scholar
10. K. Bantupalli and Y. Xie, American sign language recognition using deep learning and computer vision, in Proc. 2018 IEEE Int. Conf. Big Data, Seattle, WA, 2018, pp. 4896–4899. https://doi.org/10.1109/BIGDATA.2018.8622141 Crossref, Google Scholar
11. J. Forster, C. Schmidt, O. Koller, M. Bellgardt and H. Ney, Extensions of the sign language recognition and translation corpus RWTH-PHOENIX-weather, in Proc. Ninth Int. Conf. Language Resources and Evaluation (LREC’14), Reykjavik, Iceland, 2014, pp. 1911–1916. Google Scholar
12. S. K. Ko, C. J. Kim, H. Jung and C. Cho, Neural sign language translation based on human keypoint estimation, Appl. Sci. 9(13) (2019) 1–19. https://doi.org/10.3390/app9132683 Crossref, Google Scholar
13. S.-K. Ko, J. G. Son and H. Jung, Sign language recognition with recurrent neural network using human keypoint detection, in Proc. 2018 Conf. Research in Adaptive and Convergence Systems (ACM, 2018), pp. 326–328. https://doi.org/10.1145/3264746.3264805 Crossref, Google Scholar
14. Y. Bajaj and P. Malhotra, American sign language identification using hand trackpoint analysis, in Int. Conf. Innovative Computing and Communications (Springer, Singapore, 2022), pp. 159–171. Crossref, Google Scholar
15. M. Saerbeck and P. K. Pisharady, Gesture recognition performance score: A new metric to evaluate gesture recognition systems, https://doi.org/10.1007/978-3-319-16628-5_12. Google Scholar
16. M. A. Jayaram and H. Fleyeh, Convex hulls in image processing: A scoping review, Am. J. Intell. Syst. 6(2) (2016) 48–58. https://doi.org/10.5923/j.ajis.20160602.03 Google Scholar
17. D. Avis, D. Bremner and R. Seidel, Computational geometry how good are convex hull algorithms? Comput. Geom. 7 (1997) 265–301. Crossref, Google Scholar
18. OpenCV: Structural Analysis and Shape Descriptors’, https://docs.opencv.org/master/d3/dc0/group_imgproc_shape.html#gada4437098113fd8683c932e0567f47ba. Google Scholar
19. T. Mishra, S. S. Rao, M. P. Thanmayi and T. Bhat, Transitioning from convex hull algorithm to CNN, Int. Res. J. Eng. Technol. 7 (2020) 6202–6206. Google Scholar
20. Y. Lecun, Y. Bengio and G. Hinton, Deep learning, Nature 521(7553) (2015) 436–444. https://doi.org/10.1038/nature14539 Crossref, Google Scholar
21. S. Indolia, A. K. Goswami, S. P. Mishra and P. Asopa, Conceptual understanding of convolutional neural network — A deep learning approach, Procedia Comput. Sci. 132 (2018) 679–688. https://doi.org/10.1016/j.procs.2018.05.069 Crossref, Google Scholar
22. C. Neubauer, Evaluation of convolutional neural networks for visual recognition, IEEE Trans. Neural Netw. 9(4) (1998) 685–696. https://doi.org/10.1109/72.701181 Crossref, Google Scholar
23. G. Tolias, R. Sicre, I. Rennes and H. Jégou, Particular object retrieval with integral max-pooling of CNN activations, preprint (2015), arXiv:1511.05879. Google Scholar
24. P. C. Cattin, F. Bieder and R. Sandkühler, Comparison of methods generalizing max-and average-pooling, preprint (2021), arXiv:2103.01746. Google Scholar
25. N. Akhtar and U. Ragavendran, Interpretation of intelligence in CNN-pooling processes: A methodological survey, Neural Comput. Appl. 32(3) (2020) 879–898. https://doi.org/10.1007/s00521-019-04296-5 Crossref, Google Scholar
26. W. Shi, C. C. Loy and X. Tang, Deep specialized network for illuminant estimation, in Lecture Notes in Computer Science Subseries in Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics, Vol. 9908 (2016), pp. 371–387. https://doi.org/10.1007/978-3-319-46493-0_23 Google Scholar
27. H. Wu and X. Gu, Max-pooling dropout for regularization of convolutional neural networks. Google Scholar
28. A. Krizhevsky, I. Sutskever and G. E. Hinton, ImageNet classification with deep convolutional neural networks, Commun. ACM 60(6) (2017) 84–90. https://doi.org/10.1145/3065386 Crossref, Google Scholar
29. T. Kurita and H. Ide, Improvement of learning for CNN with ReLU activation by sparse regularization, in 2017 Int. Joint Conf. Neural Networks, Anchorage, AK, 2017. https://doi.org/10.1109/IJCNN.2017.7966185 Google Scholar
30. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever and R. R. Salakhutdinov, Improving neural networks by preventing co-adaptation of feature detectors, arXiv:1207.0580. Google Scholar
31. M. D. Zeiler and R. Fergus, Stochastic pooling for regularization of deep convolutional neural networks, preprint (2013), arXiv:1301.3557. Google Scholar
32. R. Abbasi-Asl and B. Yu, Structural compression of convolutional neural networks, preprint (2017), arXiv:1705.07356. Google Scholar
33. L. Wang, W. Hu and T. Tan, Recent developments in human motion, Pattern Recognit. 36(3) (2003) 585–601. Crossref, Google Scholar
34. C. Sminchisescu, 3D human motion analysis in monocular video techniques and challenges, in Human Motion, Computational Imaging and Vision, Vol. 36 (Springer, Dodrecht, 2008), pp. 185–211. Crossref, Google Scholar
35. J. Wang et al., Deep 3D human pose estimation: A review, Comput. Vis. Image Underst. 210 (2021) 103225. https://doi.org/10.1016/j.cviu.2021.103225 Crossref, Google Scholar
36. S. Shimada, W. Xu and C. Theobalt, Neural monocular 3D human motion capture with physical awareness, ACM Trans. Graph. 40(4) (2021) 1–15. https://doi.org/10.1145/3450626.3459825 Crossref, Google Scholar
37. Google AI Blog: MediaPipe Holistic — Simultaneous Face, Hand and Pose Prediction, on Device, https://ai.googleblog.com/2020/12/mediapipe-holistic-simultaneous-face.html. Google Scholar
38. ‘MediaPipe.pdf’. google-research/ghum at master.google-research/google-research.GitHub’, https://github.com/google-research/google-research/tree/master/ghum. Google Scholar
39. D. Chortarias, Human activity recognition with deep learning, (2021). Google Scholar
40. V. F. Rodriguez-Galiano, B. Ghimire, J. Rogan, M. Chica-Olmo and J. P. Rigol-Sanchez, An assessment of the effectiveness of a random forest classifier for land-cover classification, ISPRS J. Photogramm. Remote Sens. 67(1) (2012) 93–104. https://doi.org/ 10.1016/j.isprsjprs.2011.11.002 Crossref, Google Scholar
41. V. Bazarevsky, I. Grishchenko, K. Raveendran, T. Zhu, F. Zhang and M. Grundmann, BlazePose: On-device real-time body pose tracking, arXiv:2006.10204. Google Scholar
42. S. M. Ross, Probability, Introd. Stat. (2017) 139–201. https://doi.org/10.1016/B978-0-12-804317-2.00004-7 Crossref, Google Scholar