Research ArticleNo Access

Deep Convolution Generative Adversarial Network-Based Electroencephalogram Data Augmentation for Post-Stroke Rehabilitation with Motor Imagery

International School for Optoelectronic Engineering, Qilu University of Technology, (Shandong Academy of Sciences), Jinan 250353, P. R. China

E-mail Address: xfz@qlu.edu.cn

Search for more papers by this author

Gege Dong

International School for Optoelectronic Engineering, Qilu University of Technology, (Shandong Academy of Sciences), Jinan 250353, P. R. China

School of Electrical Engineering and Automation, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, P. R. China

Search for more papers by this author

Jincheng Li

International School for Optoelectronic Engineering, Qilu University of Technology, (Shandong Academy of Sciences), Jinan 250353, P. R. China

School of Electrical Engineering and Automation, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, P. R. China

Search for more papers by this author

Qingbo Yang

School of Mathematics and Statistics, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, P. R. China

Search for more papers by this author

Lei Wang

School of Electrical Engineering and Automation, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, P. R. China

Search for more papers by this author

Yanna Zhao

School of Information Science and Engineering, Shan Dong Normal University, Jinan 250358, P. R. China

Search for more papers by this author

Yihao Yan

International School for Optoelectronic Engineering, Qilu University of Technology, (Shandong Academy of Sciences), Jinan 250353, P. R. China

School of Electrical Engineering and Automation, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, P. R. China

Search for more papers by this author

Jinzhao Zhao

International School for Optoelectronic Engineering, Qilu University of Technology, (Shandong Academy of Sciences), Jinan 250353, P. R. China

School of Electrical Engineering and Automation, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, P. R. China

Search for more papers by this author

Shaopeng Pang

School of Electrical Engineering and Automation, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, P. R. China

E-mail Address: pang_shao_peng@163.com

Search for more papers by this author

Dongju Guo

The Department of Physical Medicine and Rehabilitation, Qilu Hospital, Cheeloo College of Medicine, Shandong University, Jinan 250012, P. R. China

E-mail Address: gdj3587@163.com

Corresponding author.

Search for more papers by this author

Yang Zhang

The Department of Physical Medicine and Rehabilitation, Qilu Hospital, Cheeloo College of Medicine, Shandong University, Jinan 250012, P. R. China

Search for more papers by this author

, and

Jiancai Leng

International School for Optoelectronic Engineering, Qilu University of Technology, (Shandong Academy of Sciences), Jinan 250353, P. R. China

E-mail Address: jiancaileng@qlu.edu.cn

Corresponding author.

Search for more papers by this author

https://doi.org/10.1142/S0129065722500393Cited by:38 (Source: Crossref)

Abstract

The motor imagery brain–computer interface (MI-BCI) system is currently one of the most advanced rehabilitation technologies, and it can be used to restore the motor function of stroke patients. The deep learning algorithms in the MI-BCI system require lots of training samples, but the electroencephalogram (EEG) data of stroke patients is quite scarce. Therefore, the expansion of EEG data has become an important part of stroke clinical rehabilitation research. In this paper, a deep convolution generative adversarial network (DCGAN) model is proposed to generate artificial EEG data and further expand the scale of the stroke dataset. First, multichannel one-dimensional EEG data is converted into a two-dimensional EEG spectrogram using EEG2Image based on the modified S-transform. Then, DCGAN is used to artificially generate EEG data based on MI. Finally, the validity of the generated artificial EEG data is proved. This paper preliminarily indicates that generating artificial stroke data is a promising strategy, which contributes to the further development of stroke clinical rehabilitation.

Keywords:

References

1. S. S. Virani, A. Alonso, E. J. Benjamin, M. S. Bittencourt, C. W. Callaway, A. P. Carson, A. M. Chamberlain, A. R. Chang, S. Cheng, F. N. Delling, L. Djousse, M. S. V. Elkind, J. F. Ferguson, M. Fornage, S. S. Khan, B. M. Kissela, K. L. Knutson, T. W. Kwan, D. T. Lackland, T. T. Lewis, J. H. Lichtman, J. H. Longenecker, M. S. Loop, P. L. Lutsey, S. S. Martin, K. Matsushita, A. E. Moran, M. E. Mussolino, A. M. Perak, W. D. Rosamond, G. A. Roth, U. K. A. Sampson, G. M. Satou, E. B. Schroeder, S. H. Shah, C. M. Shay, N. L. Spartano, A. Stokes, D. L. Tirschwell, L. B. VanWagner and C. W. Tsao, Heart disease and stroke statistics — 2020 update: A report from the American Heart Association, Circulation 141(9) (2020) e139–e596. Crossref, Medline, Web of Science, Google Scholar
2. R. Mane, T. Chouhan and C. Guan, BCI for stroke rehabilitation: Motor and beyond, J. Neural Eng. 17(4) (2020) 041001. Crossref, Medline, Web of Science, Google Scholar
3. S. H. George, M. H. Rafiei, L. Gauthier, A. Borstadd, J. A. Buforde and H. Adeli, Computer-aided prediction of extent of motor recovery following constraint-induced movement therapy in chronic stroke, Behav. Brain Res. 329 (2017) 191–199. Crossref, Medline, Web of Science, Google Scholar
4. J. Olias, R. Martín-Clemente, M. A. Sarmiento-Vega and S. Cruces, EEG signal processing in MI-BCI applications with improved covariance matrix estimators, IEEE Trans. Neural Syst. Rehabil. Eng. 27(5) (2019) 895–904. Crossref, Medline, Web of Science, Google Scholar
5. F. Xu, F. Rong, J. Leng, T. Sun, Y. Zhang, S. Siddharth and T. P. Jung, Classification of left-versus right-hand motor imagery in stroke patients using supplementary data generated by CycleGAN, IEEE Trans. Neural Syst. Rehabil. Eng. 29 (2021) 2417–2424. Crossref, Medline, Web of Science, Google Scholar
6. W. Y. Hsu, Single-trial motor imagery classification using asymmetry ratio, phase relation, wavelet-based fractal, and their selected combination, Int. J. Neural Syst. 23(2) (2013) 1350007. Link, Web of Science, Google Scholar
7. G. Rodriguez-Bermudez, P. J. Garcia-Laencina and J. Roca-Dorda, Efficient automatic selection and combination of EEG features in least squares classifiers for motor imagery brain–computer interfaces, Int. J. Neural Syst. 23(4) (2013) 1350015. Link, Web of Science, Google Scholar
8. Y. Liu, H. Zhang, M. Chen and L. Zhang, A boosting-based spatial-spectral model for stroke patients’ EEG analysis in rehabilitation training, IEEE Trans. Neural Syst. Rehabil. Eng. 24(1) (2015) 169–179. Crossref, Medline, Web of Science, Google Scholar
9. A. Ortiz-Rosario, I. Berrios-Torres, H. Adeli and J. A. Buford, Combined corticospinal and reticulospinal effects on upper limb muscles, Neurosci. Lett. 561 (2014) 30–34. Crossref, Medline, Web of Science, Google Scholar
10. A. Burns, H. Adeli and J. A. Buford, Brain–computer interface after nervous system injury, Neurosci. 20(6) (2014) 639–651. Google Scholar
11. A. Ortiz-Rosario, H. Adeli and J. A. Buford, Wavelet methodology to improve single unit isolation in primary motor cortex cells, J. Neurosci. Meth. 246 (2015) 106–118. Crossref, Medline, Web of Science, Google Scholar
12. A. Burns and H. Adeli, Wearable technology for patients with brain and spinal cord injuries, Rev. Neurosci. 28(8) (2017) 913–920. Crossref, Medline, Web of Science, Google Scholar
13. S. H. George, M. H. Rafiei, A. Borstad, H. Adeli and L. V. Gauthier, Gross motor ability predicts response to upper extremity rehabilitation in chronic stroke, Behav. Brain Res. 333 (2017) 314–322. Crossref, Medline, Web of Science, Google Scholar
14. Z. Yang, M. H. Rafiei, A. Hall, C. Thomas, H. A. Midtlien, A. Hasselbach, H. Adeli and L. V. Gauthier, A novel methodology for extracting and evaluating therapeutic movements in game-based motion capture rehabilitation systems, J. Med. Syst. 42(12) (2018) 1–14. Crossref, Web of Science, Google Scholar
15. M. H. Rafiei, K. M. Kelly, A. L. Borstad, H. Adeli and L. V. Gauthier, Predicting improved daily use of the more affected arm post-stroke following constraint-induced movement, Phys. Ther. 99(12) (2019) 1667–1678. Crossref, Medline, Web of Science, Google Scholar
16. A. Burns, H. Adeli and J. A. Buford, Upper limb movement classification via electromyographic signals and enhanced probabilistic network, J. Med. Syst. 44(10) (2020) 1–12. Crossref, Web of Science, Google Scholar
17. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville and Y. Bengio, Generative adversarial nets, Adv. Neural Inf. Process. Syst. 3 (2014) 2672–2680. Google Scholar
18. A. Radford, L. Metz and S. Chintala, Unsupervised representation learning with deep convolutional generative adversarial networks, (2015), arXiv:1511.06434. Google Scholar
19. A. B. L. Larsen, S. K. Sønderby, H. Larochelle and O. Winther, Autoencoding beyond pixels using a learned similarity metric, in Proc. Int. Conf. Mach. Learn (ICML), New York, USA (IEEE, 2016), pp. 1558–1566. Google Scholar
20. X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang and S. P. Smolley, Least squares generative adversarial networks, in Proc. 16th IEEE Int. Conf. Comput. Vis. (ICCV), Venice, Italy (IEEE, 2017), pp. 2813–2821. Crossref, Google Scholar
21. J. Y. Zhu, T. Park, P. Isola and A. A. Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks, in Proc. 16th IEEE Int. Conf. Comput. Vis. (ICCV), Venice, Italy (PMLR, 2017), pp. 2223–2232. Crossref, Google Scholar
22. M. Arjovsky, S. Chintala and L. Bottou, Wasserstein GAN, 2017, arXiv:1701.07875. Google Scholar
23. H. Zhang, I. Goodfellow, D. Metaxas and A. Odena, Self-attention generative adversarial networks, in Proc. 36th Int. Conf. Mach. Learn (ICML), Long Beach, USA (2019), pp. 7354–7363. Google Scholar
24. A. Jolicoeur-Martineau, The relativistic discriminator: A key element missing from standard GAN, 2018, arXiv:1807.00734. Google Scholar
25. M. Lucic, M. Tschannen, M. Ritter, X. Zhai and S. Gelly, High-fidelity image generation with fewer labels, in Proc. 36th Int. Conf. Mach. Learn (ICML), Long Beach, USA (PMLR, 2019), pp. 4183–4192. Google Scholar
26. F. Rong, T. Sun, F. Xu, Y. Zhang and L. Wang, Artificial topographic structure of CycleGAN for stroke patients’ motor imagery recognition, in Proc. 10th Int. Conf. Computer Engineering and Networks (CENet), Singapore (Springer, 2020), pp. 60–67. Google Scholar
27. H. Adeli, Z. Zhou and N. Dadmehr, Analysis of EEG records in an epileptic patient using wavelet transform, J. Neurosci. Meth. 123(1) (2003) 69–87. Crossref, Medline, Web of Science, Google Scholar
28. O. Faust, U. R. Acharya, H. Adeli and A. Adeli, Wavelet-based EEG processing for computer-aided seizure detection and epilepsy diagnosis, Seizure 26 (2015) 56–64. Crossref, Medline, Web of Science, Google Scholar
29. U. R. Acharya, S. L. Oh, Y. Hagiwara, J. H. Tan, H. Adeli and D. P. Subha, Automated EEG-based screening of depression using deep convolutional neural network, Comput. Meth. Prog. Biomed. 161 (2018) 103–113. Crossref, Medline, Web of Science, Google Scholar
30. U. R. Acharya, S. Vidya, S. Bhat, H. Adeli and A. Adeli, Computer-aided diagnosis of alcoholism-related EEG signals, Epilepsy Behav. 41 (2014) 257–263. Crossref, Medline, Web of Science, Google Scholar
31. V. K. Benzy, A. P. Vinod, R. Subasree, S. Alladi and K. Raghavendra, Motor imagery hand movement direction decoding using brain computer interface to aid stroke recovery and rehabilitation, IEEE Trans. Neural Syst. Rehabilitation Eng. 28(12) (2020) 3051–3062. Crossref, Medline, Web of Science, Google Scholar
32. N. M. Dawi, K. Kuca, O. Krejcar and H. Namazi, Complexity and memory-based comparison of the brain activity between ADHD and healthy subjects while playing a serious game, Fractals 29(5) (2021) 2150202. Link, Web of Science, Google Scholar
33. A. Hassanpour, M. Moradikia, H. Adeli, S. R. Khayami and P. Shamsinejadbabaki, A novel end-to-end deep learning scheme for classifying multiclass motor imagery EEG signals, Expert Syst. 36(6) (2019) e12494. Crossref, Web of Science, Google Scholar
34. G. B. Martins, J. P. Papa and H. Adeli, Deep learning techniques for recommender systems based on collaborative filtering, Expert Syst. 37(6) (2020) e12647. Crossref, Web of Science, Google Scholar
35. F. Wang, G. Song and Y. L. Mo, Shear loading detection of through-bolts in bridge structures using a percussion-based one-dimensional memory-augmented convolutional neural network, Comput.-Aided Civ. Inf. 36(3) (2021) 289–301. Crossref, Web of Science, Google Scholar
36. J. Guo, Q. Wang and Y. Li, Semi-supervised learning based on convolutional neural network and uncertainty filter for façade defects classification, Comput-Aided Civ. Inf. 36(3) (2021) 302–317. Crossref, Web of Science, Google Scholar
37. Y. A. Hsieh, Z. Yang and Y. J. Tsai, Convolutional neural network for automated classification of jointed plain concrete pavement conditions, Comput-Aided Civ. Inf. 36(11) (2021) 1382–1397. Crossref, Web of Science, Google Scholar
38. P. Wang, A. Jiang, X. Liu, J. Shang and L. Zhang, LSTM-based EEG classification in motor imagery tasks, IEEE Trans. Neural Syst. Rehabil. Eng. 26(11) (2018) 2086–2095. Crossref, Medline, Web of Science, Google Scholar
39. X. An, D. Kuang, X. Guo, Y. Zhao and L. He, A deep learning method for classification of EEG data based on motor imagery, in Proc. Int. Conf. Intelligent Computing (IEEE, Cham, 2014), pp. 203–210. Crossref, Google Scholar
40. S. M. Abdelfattah, G. M. Abdelrahman, M. Wang, Augmenting the size of EEG datasets using generative adversarial networks, in Proc. 2018 Int. Joint Conf. Neural Networks (IJCNN), (IEEE, Rio de Janeiro, Brazil, 2018), pp. 1–6. Crossref, Google Scholar
41. I. A. Corley and Y. Huang, Deep EEG super-resolution: Upsampling EEG spatial resolution with generative adversarial networks, in Proc. IEEE EMBS Int. Conf. Biomed. Informat. (BHI), (IEEE, Las Vegas, USA, 2018), pp. 100–103. Crossref, Google Scholar
42. Y. Luo and B. L. Lu, EEG data augmentation for emotion recognition using a conditional Wasserstein GAN, in Proc. 40th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. (EMBC), (IEEE, Honolulu, USA, 2018), pp. 2535–2538. Crossref, Google Scholar
43. F. Fahimi, Z. Zhang, W. B. Goh, K. K. Ang and C. Guan, Towards EEG generation using GANs for BCI applications, in Proc. IEEE EMBS Int. Conf. Biomed. Health Informat. (BHI), (IEEE, Chicago, USA, 2019), pp. 1–4. Crossref, Google Scholar
44. Y. Lee and Y. Huang, Generating target/non-target images of an RSVP experiment from brain signals in by conditional generative adversarial network, in Proc. IEEE EMBS Int. Conf. Biomed. Health Inf. (BHI), (ACM, Las Vegas, USA, 2018), pp. 182–185. Crossref, Google Scholar
45. K. G. Hartmann, R. T. Schirrmeister and T. Ball, EEG-GAN: Generative adversarial networks for electroencephalographic (EEG) brain signals (2018), arXiv:1806.01875. Google Scholar
46. S. Bhat and E. Hortal, GAN-based data augmentation for improving the classification of EEG signals, in Proc. 14th Pervasive Technologies Related to Assistive Environments Conference (PETRA), (IEEE, Corfu, Greece, 2021), pp. 453–458. Crossref, Google Scholar
47. Y. Jiao, Y. Deng, Y. Luo and B. L. Lu, Driver sleepiness detection from EEG and EOG signals using GAN and LSTM networks, Neurocomputing 408 (2020) 100–111. Crossref, Web of Science, Google Scholar
48. S. Hwang, K. Hong, G. Son and H. Byun, EZSL-GAN: EEG-based zero-shot learning approach using a generative adversarial network, in Proc. 7th Int. Winter Conf. Brain–Comput. Interface (BCI), (IEEE, Gangwon, Korea, 2019), pp. 1–4. Crossref, Google Scholar
49. S. Roy, S. Dora, K. McCreadie and G. Prasad, MIEEG-GAN: Generating artificial motor imagery electroencephalography signals, in Proc. Int. Joint Conf. Neural Netw. (IJCNN), (IEEE, Glasgow, UK, 2020), pp. 1–8. Crossref, Google Scholar
50. A. Zhang, L. Su, Y. Zhang, Y. Fu, L. Wu and S. Liang, EEG data augmentation for emotion recognition with a multiple generator conditional Wasserstein GAN, Complex Intell Syst. (2021) 1–13, https://doi.org/10.1007/s40747-021-00336-7. Medline, Web of Science, Google Scholar
51. F. Xu, W. Zhou, Y. Zhen and Q. Yuan, Classification of ECoG with modified S-transform for brain–computer interface, Int. J. Comput. Intell. Syst. 10(18) (2014) 8029–8041. Google Scholar
52. H. S. Nogay and H. Adeli, Detection of epileptic seizure using pretrained deep convolutional neural network and transfer learning, Eur. Neurol. 83 (2020) 602–614. Crossref, Medline, Web of Science, Google Scholar
53. R. G. Stockwell, L. Mansinha and R. P. Lowe, Localization of the complex spectrum: The S transform, IEEE Trans. Signal Process. 44(4) (1996) 998–1001. Crossref, Web of Science, Google Scholar
54. S. Siddharth, T. P. Jung and T. J. Sejnowski, Utilizing deep learning towards multi-modal bio-sensing and vision-based affective computing, IEEE Trans. Affect Comput. 13(1) (2019) 96–107. Crossref, Web of Science, Google Scholar
55. Y. Yao, J. Plested and T. Gedeon, A feature filter for EEG using Cycle-GAN structure, in Proc. Int. Conf. Neural Inform. Proces. (ICONIP), (Springer, Cham, 2018), pp. 567–576. Crossref, Google Scholar
56. J. Viola, Y. Q. Chen and J. Wang, FaultFace: Deep convolutional generative adversarial network (DCGAN) based ball-bearing failure detection method, Inf. Sci. 542 (2021) 195–211. Crossref, Web of Science, Google Scholar
57. Q. Zhang and Y. Liu, Improving brain computer interface performance by data augmentation with conditional deep convolutional generative adversarial networks, (2018), arXiv:1806.07108. Google Scholar