Gesture recognition plays an important role in human–computer interaction. However, most existing methods are complex and time-consuming, which limit the use of gesture recognition in real-time environments. In this paper, we propose a static gesture recognition system that combines depth information and skeleton data to classify gestures. Through feature fusion, hand digit gestures of 0–9 can be recognized accurately and efficiently. According to the experimental results, the proposed gesture recognition system is effective and robust, which is invariant to complex background, illumination changes, reversal, structural distortion, rotation, etc. We have tested the system both online and offline which proved that our system is satisfactory to real-time requirements, and therefore it can be applied to gesture recognition in real-world human–computer interaction systems.

Keywords:

References

1. A. Erol, G. Bebis and M. Nicolescu, Vision-based hand pose estimation: A review, Comput. Vis. Image Underst. 108 (1) (2007) 52–73. Crossref, Web of Science, Google Scholar
2. K. M. Sagayam and D. J. Hemanth, Virtual Reality: Hand Posture and Gesture Recognition Techniques for Virtual Reality Applications: A Survey (Springer-Verlag, 2016), pp. 1–17. Google Scholar
3. S. Reifinger, F. Wallhoff, M. Ablassmeier, T. Poitschke and G. Rigoll, Static and dynamic hand-gesture recognition for augmented reality applications, in ACM Int. Conf. Intelligent Multimodal Interaction Environments (ACM Press, Beijing, 2007), pp. 728–737. Google Scholar
4. L. Riouxmaldague and P. Giguere, Sign language fingerspelling classification from depth and color images using a deep belief network, in Canadian Conf. Computer and Robot Vision (CRV) (IEEE Press, Montreal, 2014), pp. 92–97. Google Scholar
5. G. Diraco, A. Leone and P. Siciliano, An active vision system for fall detection and posture recognition in elderly healthcare, in Europe Conf. Design, Automation and Test in Europe Conference and Exhibition (DATE) (IEEE Press, Dresden, 2010), pp. 1536–1541. Google Scholar
6. W. Gao, G. Fang and D. Zhao, A Chinese sign language recognition system based on SOFM/SRN/HMM, Pattern Recogn. 37 (12) (2004) 2389–2402. Crossref, Web of Science, Google Scholar
7. B. Y. L. Li, A. S. Mian, W. Liu and A. Krishna, Face recognition based on Kinect, Pattern Anal. Appl. 19 (4) (2015) 977–987. Crossref, Web of Science, Google Scholar
8. G. Goswami, S. Bharadwaj and M. Vatsa, On RGB-D face recognition using Kinect, in IEEE Int. Conf. Biometrics: Theory, Applications and Systems (BTAS) (IEEE Press, Arlington, 2013), pp. 1–6. Google Scholar
9. M. Morana, 3D-scene reconstruction using Kinect, Advances onto the Internet of Things (Springer-Verlag, 2014), pp. 179–190. Crossref, Google Scholar
10. Y. Li, Hand gesture recognition using Kinect, in IEEE Int. Conf. Software Engineering and Service Science (ICSESS) (IEEE Press, Beijing, 2012), pp. 196–199. Google Scholar
11. L. Bretzner, I. Laptev and T. Lindeberg, Hand gesture recognition using multi-scale colour features, hierarchical models and particle filtering, in IEEE Int. Conf. Automatic Face and Gesture Recognition (AFGR) (IEEE Press, Washington, 2002), pp. 423–428. Google Scholar
12. C. Keskin, F. Kirac, Y. E. Kara and L. Akarun, Real-time hand pose estimation using depth sensors, in IEEE Int. Conf. Computer Vision Workshops (ICCV Workshops) (IEEE Press, Barcelona, 2011), pp. 1228–1234. Google Scholar
13. Z. Mo and U. Neumann, Real-time hand pose recognition using low-resolution depth images, in IEEE Int. Conf. Computer Vision and Pattern Recognition (CVPR) (IEEE Press, New York, 2006), pp. 1499–1505. Google Scholar
14. S. Belongie, J. Malik and J. Puzicha, Shape matching and object recognition using shape contexts, Pattern Anal. Mach. Intell. 24 (4) (2002) 509–522. Crossref, Web of Science, Google Scholar
15. X. Bai and L. J. Latecki, Path similarity skeleton graph matching, Pattern Anal. Mach. Intell. 30 (7) (2008) 1282–1292. Crossref, Web of Science, Google Scholar
16. H. Jinguang, G. Award and A. Sutherland, Automatic skin segmentation and tracking in sign language recognition, IET Comput. Vis. 3 (1) (2009) 24–35. Crossref, Web of Science, Google Scholar
17. F. Parvini, D. Mcleod, C. Shahabi et al., Int. Conf. Human Computer Interaction: An Approach to Glove-Based Gesture Recognition (Springer-Verlag, 2009), pp. 236–245. Google Scholar
18. Z. Ren, J. Yuan and J. Meng, Robust part-based hand gesture recognition using Kinect sensor, IEEE Trans. Multimed. 15 (5) (2013) 1110–1120. Crossref, Web of Science, Google Scholar
19. C. Keskin, F. Kirac, Y. E. Kara and L. Akarun, Randomized decision forests for static and dynamic hand-shape classification, in IEEE Int. Conf. Computer Vision and Pattern Recognition Workshops (CVPRW) (IEEE Press, Providence, 2012), pp. 31–36. Google Scholar
20. F. Kirac, Y. E. Kara and L. Akarun, Hierarchically constrained 3D hand pose estimation using regression forests from single frame depth data, Pattern Recogn. Lett. 50 (2014) 91–100. Crossref, Web of Science, Google Scholar
21. M. J. Jones and J. M. Rehg, Statistical color models with application to skin detection, Computr. Vis. Pattern Recogn. 1 (1) (1999) 274–280. Google Scholar
22. W. Ying, J. Lin and T. S. Huang, Capturing natural hand articulation, in IEEE Int. Conf. Computer Vision (IEEE Press, Vancouver, 2001), pp. 426–432. Google Scholar
23. J. Lin, W. Ying and T. S. Huang, 3D model-based hand tracking using stochastic direct search method, in IEEE Int. Conf. Automatic Face and Gesture Recognition (AFGR) (IEEE Press, Seoul, 2004), pp. 693–698. Google Scholar
24. C. Chuqing, L. Ruifeng and G. Lianzheng, Real-Time Multi-Hand Posture Recognition, in IEEE Int. Conf. Computer Design and Applications (ICCDA) (IEEE Press, Qinhuangdao, 2010), pp. 619–623. Google Scholar
25. M. K. Bhuyan, D. R. Neog and M. K. Kar, Fingertip detection for hand pose recognition, Int. J. Comput. Sci. Engrg. 4 (3) (2012) 501–511. Google Scholar
26. D. D. Nguyen, T. C. Pham, X. D. Pham, S. H. Jin and J. W. Jeon, Finger extraction from scene with grayscale morphology and BLOB analysis, in IEEE Int. Conf. Robotics and Biomimetics (IEEE Press, Bangkok, 2009), pp. 324–329. Google Scholar
27. C. Chinchen, L. Chengyi and T. Wenkai, Feature alignment approach for hand posture recognition based on curvature scale space, Neurocomputing 71 (10) (2008) 1947–1953. Web of Science, Google Scholar
28. S. P. Priyal and P. K. Bora, A robust static hand gesture recognition system using geometry-based normalizations and Krawtchouk moments, Pattern Recogn. 46 (8) (2013) 2202–2219. Crossref, Web of Science, Google Scholar
29. R. Lienhart and J. Maydt, An extended set of Haar-like features for rapid object detection, in IEEE Int. Conf. Image Processing (IEEE Press, Rochester, 2002), pp. 900–903. Google Scholar
30. J. Zondag, T. Gritti and V. Jeanne, Practical study on real-time hand detection, in IEEE Int. Conf. Affective Computing and Intelligent Interaction and Workshops (IEEE Press, Amsterdam, 2009), pp. 1–8. Google Scholar
31. Y. Zhou, G. Jiang and Y. Lin, A novel finger and hand pose estimation technique for real-time hand gesture recognition, Pattern Recogn. 49 (2016) 102–114. Crossref, Web of Science, Google Scholar
32. X. Suau, M. Alcoverro, A. Lopezmendez, J. Ruizhidalgo and J. R. Casas, Real-time fingertip localization conditioned on hand gesture classification, Image Vis. Comput. 32 (8) (2014) 522–532. Crossref, Web of Science, Google Scholar
33. M. Tang, Recognizing hand gestures with Microsoft’s Kinect, Thesis (Department of Electrical Engineering, Stanford University, 2011). Google Scholar
34. J. Suarez and R. R. Murphy, Hand gesture recognition with depth images: A review, in IEEE Int. Conf. Robot and Human Interactive Communication (IEEE Press, Paris, 2012), pp. 411–417. Google Scholar
35. C. Zhang and Y. Tian, Histogram of 3D facets: A depth descriptor for human action and hand gesture recognition, in IEEE Int. Conf. Computer Vision and Image Understanding (IEEE Press, Shanghai, 2013), pp. 29–39. Google Scholar
36. Y. Fang, H. Liu, G. Li and X. Zhu, A multichannel surface EMG system for hand motion recognition, Int. J. Hum. Robot. 12 (2) (2015) 1550011. Link, Web of Science, Google Scholar
37. P. K. Pisharady, P. Vadakkepat and L. A. Poh, Hand posture and face recognition using a fuzzy-rough approach, Int. J. Hum. Robot. 7 (3) (2014) 63–80. Google Scholar
38. D. G. Lowe, Distinctive image features from scale-invariant keypoints, Int. J. Comput. Vis. 60 (2) (2004) 91–110. Crossref, Web of Science, Google Scholar
39. Z. Ju and H. Liu, Human hand motion analysis with multisensory information, IEEE/ASME Trans. Mechatronics, 19 (2) (2014) 456–466. Crossref, Web of Science, Google Scholar
40. Z. Ju, D. Gao, J. Cao and H. Liu, A novel approach to extract hand gesture feature in depth images, Multimedia Tools Appl. 75 (2016) 11929. Crossref, Web of Science, Google Scholar
41. Z. Ju, X. Ji, J. Li and H. Liu (In Press) An integrative framework for human hand gesture segmentation in RGB-D data, IEEE Syst. J., Doi: 10.1109/JSYST.2015.2468231. Google Scholar