Pattern RecognitionNo Access

A Combined Motion-Audio School Bullying Detection Algorithm

Department of Information and Communication Engineering, Harbin Institute of Technology, No. 2 Yikuang Street, Harbin 150080, P. R. China

Health and Wellness Measurement Research Group, OPEM Unit, University of Oulu, Pentti Kaiteran Katu 1, Oulu 90014, Finland

E-mail Address: yeliang@hit.edu.cn

Corresponding author.

Search for more papers by this author

Peng Wang

China Electronics Technology Group Corporation, No. 8 Guorui Road, Nanjing 210012, P. R. China

E-mail Address: wphitstudent@163.com

Search for more papers by this author

Le Wang

Department of Information and Communication Engineering, Harbin Institute of Technology, No. 2 Yikuang Street, Harbin 150080, P. R. China

E-mail Address: 1659412561@qq.com

Search for more papers by this author

Hany Ferdinando

Health and Wellness Measurement Research Group, OPEM Unit, University of Oulu, Pentti Kaiteran Katu 1, Oulu 90014, Finland

Department of Electrical Engineering, Petra Christian University, Siwalankerto 121 - 131, Surabaya 60236, Indonesia

E-mail Address: hferdina@ee.oulu.fi

Search for more papers by this author

Tapio Seppänen

Physiological Signal Analysis Team, University of Oulu, Pentti Kaiteran Katu 1, Oulu 90014, Finland

E-mail Address: tapio@ee.oulu.fi

Search for more papers by this author

, and

Esko Alasaarela

Health and Wellness Measurement Research Group, OPEM Unit, University of Oulu, Pentti Kaiteran Katu 1, Oulu 90014, Finland

E-mail Address: esko.alasaarela@ee.oulu.fi

Search for more papers by this author

https://doi.org/10.1142/S0218001418500465Cited by:17 (Source: Crossref)

Abstract

School bullying is a common social problem, which affects children both mentally and physically, making the prevention of bullying a timeless topic all over the world. This paper proposes a method for detecting bullying in school based on activity recognition and speech emotion recognition. In this method, motion and voice data are gathered by movement sensors and a microphone, followed by extraction of a set of motion and audio features to distinguish bullying incidents from daily life events. Among extracted motion features are both time-domain and frequency-domain features, while audio features are computed with classical MFCCs. Feature selection is implemented using the wrapper approach. At the next stage, these motion and audio features are merged to form combined feature vectors for classification, and LDA is used for further dimension reduction. A BPNN is trained to recognize bullying activities and distinguish them from normal daily life activities. The authors also propose an action transition detection method to reduce computational complexity for practical use. Thus, the bullying detection algorithm will only run, when an action transition event has been detected. Simulation results show that the combined motion-audio feature vector outperforms separate motion features and acoustic features, achieving an accuracy of 82.4% and a precision of 92.2%. Moreover, with the action transition method, the computation cost can be reduced by half.

Keywords:

References

1. S. Aino, A. Esko, S. Hannu et al., A two-threshold fall detection algorithm for reducing false alarms, 2012 6th Int. Symp. Medical Information and Communication Technology, (2012), pp. 1–4. Google Scholar
2. S. Chernbumroong, S. Cang and H. Yu, Genetic algorithm-based classifiers fusion for multisensor activity recognition of elderly people, IEEE J. Biomed. Health Inform. 19 (1) (2015) 282–289. Web of Science, Google Scholar
3. W. Chung-Hsien and L. Wei-Bin, Emotion recognition of affective speech based on multiple classifiers using acoustic-prosodic information and semantic labels, IEEE Trans. Affective Comput. 2 (1) (2011) 10–21. Web of Science, Google Scholar
4. Coordination and cooperation in integrated child protection systems, The Ninth European Forum on the Rights of the Child, 30 April 2015. Google Scholar
5. J. Deng, X. Xu, Z. Zhang et al., Semisupervised autoencoders for speech emotion recognition, IEEE/ACM Trans. Audio Speech Language Process. 26 (1) (2018) 31–43. Google Scholar
6. A. D. Dileep and C. C. Sekhar, GMM-based intermediate matching kernel for classification of varying length patterns of long duration speech using support vector machines, IEEE Trans. Neural Netw. Learning Syst. 25 (8) (2014) 1421–1432. Web of Science, Google Scholar
7. G.-C. Enrique and F. B. Ramon, An improved three-stage classifier for activity recognition, Int. J. Pattern Recognit. Artif. Intell. 32 (1) (2018) 1860003-1–18. Web of Science, Google Scholar
8. H. Ferdinando, L. Ye, T. Han et al., Violence detection from ECG signals: A preliminary study, J. Pattern Recognit. Res. 12 (1) (2017) 7–18. Google Scholar
9. H. Ferdinando, L. Ye, T. Seppänen et al., Emotion recognition by heart rate variability, Australian J. Basic Appl. Sci. Special 8 (14) (2014) 50–55. Google Scholar
10. H. Ferdinando, T. Seppänen and E. Alasaarela, Enhancing emotion recognition from ECG signals using supervised dimensionality reduction, in Proc. 6th Int. Conf. Pattern Recognition Applications and Methods (ICPRAM) (2017), pp. 112–118. Google Scholar
11. G. Hache, E. D. Lemaire and N. Baddour, Wearable mobility monitoring using a multimedia smartphone platform, IEEE Trans. Instrum. Meas. 60 (9) (2011) 3153–3161. Web of Science, Google Scholar
12. W. Kunxia, A. Ning, L. Bing Nan et al., Speech emotion recognition using Fourier parameters, IEEE Trans. Affective Comput. 6 (1) (2015) 69–75. Web of Science, Google Scholar
13. Y. Liu, S. C. H. Subramaniam, O. Sourina et al., EEG-based mental workload and stress recognition of crew members in maritime virtual simulator: A case study, 2017 Int. Conf. Cyberworlds (CW) (2017), pp. 64–71. Google Scholar
14. R. Lun and W. Zhao, A survey of applications and human motion recognition with Microsoft Kinect, Int. J. Pattern Recognit. Artif. Intell. 29 (5) (2015) 1–49. Link, Web of Science, Google Scholar
15. D. Z. Marius and M. F. Silvia, A study about the automatic recognition of the anxiety emotional state using Emo-DB, 2015 E-Health and Bioengineering Conference (EHB) (2015), pp. 1–4. Google Scholar
16. A. Mohanta and V. K. Mittal, Classifying emotional states using pitch and formants in vowel regions. 2016 Int. Conf. Signal Processing and Communication (ICSC) (2016), pp. 458–463. Google Scholar
17. P. D. Prajakta, S. Kailash and P. Malathi, Speaker dependent speech emotion recognition using MFCC and support vector machine, 2016 Int. Conf. Automatic Control and Dynamic Optimization Techniques (ICACDOT) (2016), pp. 1080–1084. Google Scholar
18. L. Qiang, H. Qi and S. Limin, Collaborative recognition of queuing behavior on mobile phones, IEEE Trans. Mobile Comput. 15 (1) (2016) 60–73. Web of Science, Google Scholar
19. K. Scherer, Vocal communication of emotion: A review of research paradigms, Speech Commun. 40 (1–2) (2003) 227–256. Web of Science, Google Scholar
20. J. Sharon, Bullying survey: Most teens have hit someone out of anger, USA Today, 26 October 2010. Google Scholar
21. P. Song and W. Zheng, Feature selection based transfer subspace learning for speech emotion recognition, IEEE Trans. Affective Comput. (2018) 1–11. Web of Science, Google Scholar
22. Y. Sung-Nien and C. Shu-Feng, Emotion state identification based on heart rate variability and genetic algorithm, Engineering in Medicine and Biology Society (EMBC). 2015 37th Annual International Conference of the IEEE (2015), pp. 538–541. Google Scholar
23. I. Theodoros and P. Georgios, Using an automated speech emotion recognition technique to explore the impact of bullying on pupils social life, 2011 Panhellenic Conf. Informatics (2011), pp. 18–22. Google Scholar
24. L. Vanitha and G. R. Suresh, Hybrid SVM classification technique to detect mental stress in human beings using ECG signals, 2013 Int. Conf. Advanced Computing and Communication Systems (2013), pp. 1–6. Google Scholar
25. L. Wanyu and S. Dandan, Analysis of school violence in Japan, Heihe J. 6 (2011) 204–205. Google Scholar
26. Y. Xizhe, S. Weiming, J. Samarabandu et al., Human activity detection based on multiple smart phone sensors and machine learning algorithms, IEEE 19th Int. Conf. Computer Supported Cooperative Work in Design (CSCWD) (2015), pp. 582–587. Google Scholar
27. C. C. Yang and Y. L. Hsu, A review of accelerometry-based wearable motion detectors for physical activity monitoring, Sensors 10 (2010) 7772–7788. Web of Science, Google Scholar
28. L. Ye, H. Ferdinando, T. Seppänen et al., An instance-based physical violence detection algorithm for school bullying prevention, 2015 International Wireless Communications and Mobile Computing Conference (IWCMC) (2015), pp. 1384–1388. Google Scholar
29. L. Ye, H. Ferdinando, T. Seppänen et al., Physical violence detection for preventing school bullying, Adv. Artif. Intell. 2014 (2014) 1–9. Google Scholar
30. G. Yongbin, L. Hyo Jong and R. M. Mehmood, Deep learning of EEG signals for emotion recognition, 2015 IEEE Int. Conf. Multimedia & Expo Workshops (ICMEW) (2015), pp. 1–5. Google Scholar
31. H. Yongming, W. Ao, Z. Guobao et al., Extraction of adaptive wavelet packet filter-bank- based acoustic feature for speech emotion recognition, IET Signal Process. 9 (4) (2015) 341–348. Web of Science, Google Scholar