Online and Cross-User Finger Movement Pattern Recognition by Decoding Neural Drive Information from Surface Electromyogram

1. Introduction

Brain–computer interfaces (BCIs) are a new method of communication and control between a neural system and devices. In the past 10 years, BCIs have been successfully implemented in rehabilitation techniques for neural system injuries¹ such as stroke,^2,3 spinal cord injury,⁴ and other injuries.^5,6 There is substantial demand for natural and effective BCI in the field of human–machine interaction,⁷ but there are major challenges in precisely decoding motor intentions from biological signals collected from the brain, peripheral nerves, and muscles.⁸

One of the important issues for BCIs is precise pattern recognition of dexterous finger movement from surface electromyography (SEMG) signals,⁹ which has significant potential in applications related to industrial robots,¹⁰ prostheses¹¹ and assistive devices.¹² SEMG signal is an electrophysiological signal that are collected from the surface of the skin.¹³ Myoelectric control based on SEMG is one of the most used interfaces due to its noninvasiveness.^14,15 Furthermore, neural drive information contained in SEMG signals can help to decode users’ intentions based on the neurophysiology of human movements.^16,17,18 Numerous studies have demonstrated good performance of myoelectric control in gesture recognition, especially for the recognition of dexterous finger movement. In addition, many efforts have been devoted to improving its precision and robustness.^19,20,21

Conventional myoelectric pattern recognition methods primarily relied on global features in the time or frequency domain that can be directly reflected from the SEMG signals.^22,23 These features served as input in advanced machine learning or deep learning networks. Although high performance can be achieved using global SEMG features and data-driven models, the correlation between SEMG and movement has remained a “black box” because those features can only capture limited information about the neural activities of the recorded electrodes.²⁴

With recent developments in high-density SEMG (HD SEMG) measurements and instrumentation, SEMG decomposition has been extensively investigated as a way to resolve SEMG data into motor unit (MU) action potential trains (MUAPTs),^25,26,27 which are the basic components of the peripheral neuromuscular system.^16,17,18 This technique has made it possible to noninvasively decode the ”ultimate neural code” of human movements following physiological principles.²⁹ Representative decomposition methods include convolution kernel compensation (CKC),³⁰ progressive FastICA peel-off (PFP),³¹ and other FastICA-based decomposition methods.^32,33,34 In addition, online versions of CKC and PFP have been developed for actual implementation.^35,36,37,38

The development of SEMG decomposition is considered a revolution in SEMG signal processing and has changed the perspective from a macroscopic view of stochastic noise carrying activation information to a microscopic view of signal containing tens of neural sources, thus making it possible to decode neural drive information noninvasively.²⁴ Farina et al. established a man/machine interface that took advantages of MUs’ discharges.³⁹ They also explored the application potential of neural decoding approach using automatic and real-time electromyogram (EMG) decomposition.⁴⁰ Zhao et al. decoded movement patterns by mining microscopic MU information and mapping it to movements.⁴¹ Montazerin et al. proposed a compact Transformer-based framework for hand gesture recognition utilizing MU discharge information.⁴² Real-time myoelectric applications have also been developed based on the neural decoding approach. For example, Chen et al. recognized hand gestures across multiple motor tasks by decoding MU discharges in real time.⁴³ Meng and Hu investigated the possibilities of utilizing deep learning to online extract neural drive information and predict multi-finger forces.⁴⁴

However, these methods were all developed based on laboratory conditions specific to individual users. Due to the physiological variations across different users, the performance can be greatly compromised when data from a new user are directly fed into a pre-trained network, which limits practical usage.^45,46 To address the problem of cross-user robustness in myoelectric control, transfer learning (TL) techniques have been extensively investigated.^47,48,49 For example, the unsupervised domain adaptation (UDA) strategy has attracted much interest to adapt to cross-user conditions. The aim of UDA is to improve the capability of the model to effectively recognize target domain samples in the absence of labeled data,⁵⁰ which has shown success in cross-user myoelectric pattern recognition. Zhang et al. designed a self-guided adaptive sampling (SGAS) strategy to screen out reliable instantaneous samples to updating the classifier.⁵¹ Guo et al. introduced a novel algorithm called the locality preserving and maximum margin criterion (LPMM) by integrating a main domain alignment module, pseudo-label selection module, and iteration result selection module.⁵² Li et al. proposed a novel optimal transport assisted student–teacher framework (OT-ST) to facilitate transfer across user domains.⁵³ However, current studies on cross-user myoelectric pattern recognition still approached the problem at the macroscopic level of global feature extraction and have neglected the microscopic information of neural control. Compared with global features, MU activities obtained from SEMG decomposition contain more fine-grained neural drive information, and a physiological solution can be found for cross-user conditions. It is still a great challenge to determine a way to appropriately and comprehensively use the microscopic information from decomposed MUs toward robust myoelectric control system.

With all the considerations above, a novel method for myoelectric pattern recognition is proposed to enhance its robustness against cross-user conditions. This paper is the first to develop a UDA-based framework driven by individual MU activities. The proposed method referred to our previous study of OT-ST⁵³ and made specific modifications to integrate the neural decoding approach. The neural decoding approach was employed to mine more fine-grained microscopic information and establish the correspondence between MUs and movement patterns. Conventional global SEMG features were replaced with microscopic features extracted from the decomposed MUAPTs. The OT-ST model did not require special repetitive training or any labeled data for calibration, and can incrementally learn from new testing samples and therefore precisely classify the MUs on the unlabeled data. Furthermore, the final decisions were made by voting on the results of the classification of all activated MUs. Our method can realize robust cross-user gesture recognition at a microscopic level and has potential applications in intelligent neural interfaces and biomedical engineering.

2. Methods

2.1. Subjects

An experiment was performed involving 15 subjects (S1–S15, including 11 males and 4 females aged $23.40\pm 1.35$ years). All the participants were healthy and had no known history of any neuromuscular injuries or disorders. Prior to the experiments, each subject provided informed written consent to the participant in accordance with the protocols approved by the Ethics Review Committee of the University of Science and Technology of China (Hefei, Anhui, China).

2.2. Experimental protocols

A flexible electrode array was attached to the finger extensor muscles of the dominant hand of each subject to record HD-SEMG signals (Fig. 1).

Electrodes were arranged in a $16\times 8$ array. The diameter of each electrode probe was 3mm and the inter-electrode distance was 8mm. Multi-channel HD-SEMG signals were recorded using a reliable, homemade data acquisition system that has been validated in our previous studies.^41,53,54,55 A 16-bit analog-to-digital converter was used to digitize SEMG signals, and digitalization was conducted in a chronological sequence with a sample rate of 2kHz.

Before the experiment, the skin of all subjects was cleaned with alcohol and then the electrode array was applied. Seven finger movement tasks (denoted as F1–F7) were performed and involved maintaining a natural medium level of strength for at least 5s. In each trial, each subject performed movement tasks F1–F7 in a random order, as shown by the example in Fig. 2. To avoid the possible effects of fatigue, a resting period was required between every two trials. Five trials were conducted for each subject, and the experimental protocol involved 525 trials (15 subjects $\times 7$ tasks $\times 5$ trials).

Before feature extraction, some preprocessing procedures were implemented in MATLAB software to minimize possible interference. A 10th-order Butterworth bandpass filter with a bandwidth of 20–500Hz was first employed to reduce possible high- or low-frequency noise artifacts from surrounding electrical equipment. Additionally, a 50Hz second-order notch filter was also used to eliminate power line interference.

2.3. Feature extraction from MU activities

For each subject, in the source domain, SEMG signals were decomposed into MU spike trains (MUSTs) through an offline automatic PFP (APFP) method.⁵⁶ The APFP method has the ability to decompose HD-SEMG signals automatically and does not need any manual interaction. Briefly, the decomposition process of APFP is first implemented through the classical FastICA algorithm,⁵⁷ which consists of the extension and whitening of the multi-channel SEMG signals and the iteration of a fix-point algorithm.⁵⁸ This process produces a set of separation vectors and corresponding decomposed source signals that are expected to carry MU information. An initial MUST can be extracted from a source signal. However, the spikes in the initial MUST from one source signal often do not belong to just one MU due to heavy MUAP superimposition or severe MU firing synchronization. Thus, a valley-seeking clustering approach⁵⁹ is used to distinguish these spikes and then the constrained FastICA algorithm⁶⁰ is performed using the extracted and clustered spike train as constraints to correct possible MU discharge errors. Finally, a “peel-off” procedure is performed to subtract the obtained MU activities from the original signals. More MUs can emerge when processing the residual signals again with the FastICA algorithm. The details of the APFP method can be found in our previous studies.^31,56,61

After the decomposition process of the APFP, we divided the obtained MUSTs into a series of temporally overlapping windows. The window length and stride were 1s and 0.2s, respectively.^37,38 MUs with more than 10 discharges in a single time window were regarded as activated ones and their MUAP waveforms in two-dimensional distributions were estimated through a spike-triggered averaging (STA) algorithm.⁶² To estimate the MUAP waveform, 30-ms segments (15 ms before and after the discharge timings) were extracted and then averaged. The multi-channel MUAPTs can be calculated from the convolution model.^30,31 Based on our previous study,⁵³ five features were used: two from time-dependent parameters (root mean square (RMS) and wavelength (WL)), two from time-dependent power spectrum descriptors (f₁ and f${}_6)$,${}^{63}$ and sample entropy. Therefore, each multi-channel MUAPT has a $16\times 8\times 5$ feature matrix, which served as basic samples in the form of a tensor. Figure 3 shows the process of feature extraction process from MU activities.

While testing the model in the target domain, we used an online PFP method to acquire MU activities in real time.³⁷ To obtain enough separation vectors for online SEMG decomposition, one trial with the target domain was used to calculate separation vectors, and four other trials were used for classification. The details of online SEMG decomposition can be found in our previous study.³⁷ Briefly, the steps are as follows:

	Step 1. Calculate the separation vectors using a trial of SEMG data.
	Step 2. For each time window of the SEMG data stream, directly multiply the separation vectors and the extended and whitened data to obtain a series of MU source signals.
	Step 3. Extract the MUSTs from the source signals using a successive multi-threshold algorithm.
	Step 4. Connect the MUSTs over the sliding time windows.
	Step 5. If no new data are input, stop the decomposition process and output all the MUSTs.

For the sake of consistency with the training data, the length and stride of the time window in online SEMG decomposition were also 1s and 0.2s. Then the features in the target domain were extracted through the steps above. By using online SEMG decomposition, the MUSTs can be precisely identified online from the SEMG data stream.³⁷ MU discharge events with 80% overlapping within a single time window were used to ensure the continuity of the decomposition. The corresponding MUAPTs can be obtained through the decomposition steps, and the features are then extracted.

2.4. Finger movement recognition

Figure 4 shows a whole block diagram of the proposed cross-user myoelectric pattern recognition method. The teacher and student models were established with the same structure. Three conv layers, batch normalization layers and ReLu layers were designed for the model. Each conv layer had a kernel size of $3\times 3$, stride of $1\times 1$, and 64 filters. A max-pooling layer with a filter size of $2\times 2$ and a stride of 2 was also used.^41,53 Finally, a flatten layer and a full connection layer were employed to classify the decomposed MUs.

We used the stochastic gradient descent (SGD) algorithm⁶⁴ for training, cross-entropy loss was used as the training loss. The training process was completed after 10 epochs, and the batch size was set 24.

During network testing, the features of online decomposed MUs were sequentially input into the trained models in batches. All the processed input batches were used as calibration data, while the entire target domain dataset served as test data. The teacher model processed these samples to generate corresponding outputs. Normalized classification scores $P^E\in R^{B\times C}$ were obtained using a soft-max layer.

The teacher model simultaneously generated pseudo labels to teach the student model to adaptively update for each batch. Next, the model adaptation step was initiated. An OT algorithm⁶⁵ was applied to provide pseudo labels against the bias problem, and its effectiveness has been proved in our previous work.⁵³ B is the sample number in a single batch, and C is the number of movement patterns (7 in this study). The OT algorithm aims to find optimal soft pseudo labels $Q\in R^{B\times C}$ :

where $H(Q)$ denotes the entropy regularization term and its impact is controlled by $\lambda$. The classification score $P^E$ serves as a cost matrix. Then, we can obtain the optimal soft pseudo labels $Q_{\mathrm{\text{soft}}}$ :

where u and v are normalization parameters as computed in sink-horn. The corresponding hard pseudo labels $\hat{Q}$ were calculated as where ${\hat{Q}}_{i,j}$ indicated the ith sample labeled as the hard pseudo label value of pattern j.

These hard pseudo labels were used for model adaption through cross-entropy loss between predicted results and pseudo labels. The OT algorithm was also applied to update the output of the student model for MU classification. Testing data were continuously input, and the model was adaptively updated to correctly classify MUs, which made it straightforward to classify a single MU as one movement pattern for each testing sample. However, co-activation among multiple movement patterns should also be considered for the final decision about finger movement.

To describe the many-to-many correspondence between MUs and movement patterns, we employed a fuzzy weighted decision (FWD) technique.⁴¹ According to muscle physiology, the firing frequency of each MU can reflect the activation level and corresponding strength of the neural drive. We calculated the number of discharges within each time window for each MU and used it as a weight for the MU. Thus, the probability $P_{i,c}$ from the student model of the ith MU was weighted by the number of firings to evaluate the contribution of the ith MU to the movement pattern c. The final movement decision in a single time window can then be obtained :

where $ĉ$ denotes the predicted pattern. $P_{i,c}$ is the output of student model of ith MU to the pattern c. $N_i$ represents the discharge number of ith MU in the window. n is the number of activated MUs in the window. By using the FWD algorithm, the contributions of each MU to the movement patterns in a single time window were calculated to finally determine the movement pattern. This process was named the MU FWD voting strategy and is illustrated in Fig. 5.

2.5. Performance evaluation

We applied a 15-fold cross-validation strategy to comprehensively assess the performance of the proposed method. SEMG data from 14 subjects were used as a training dataset in the source domain and the data from the remaining subject were used as a testing dataset in the target domain. The MU classification accuracy was calculated for each batch across all testing samples.

To evaluate the effectiveness of MU classification, an MU-based baseline method without the teacher model was employed for comparison. In addition, we compared the proposed method with several state-of-the-art myoelectric domain adaption methods that use EMG global features (termed as EMG methods): SGAS,⁵¹ LPMM⁵² and OT-ST.⁵³ Unlike the proposed MU-based method, the output from the network of the EMG methods is the final decision of movement without an MU FWD voting strategy. The parameters of the comparison EMG methods were all consistent with the relevant literature.^51,52,53

For online testing, a data recording system was connected to a desktop computer with an Intel core i9 CPU, 32 GB of RAM and an RTX3080Ti GPU, which was used for data transfer, extraction of MU activities and pattern recognition. All tasks were implemented in the PyTorch framework.

2.6. Statistical analysis

We performed statistical analyses to evaluate the accuracy of both MU classification (using the MU-based baseline and the proposed methods) and the finger movement recognition (SGAS, LPMM, OT-ST, and the proposed method). The normality of the data distribution was tested using the Shapiro–Wilk test, and the sphericity was tested by Mauchly’s test for comparisons involving at least three groups. For groups that satisfied the assumptions of normality and sphericity, a parametric analysis was performed using a repeated-measures analysis of variance (ANOVA) for multi-group comparisons. Otherwise, a nonparametric analysis approach was performed using the Wilcoxon signed-rank test or Friedman test. When applicable, after the ANOVA or Friedman tests, Holm–Bonferroni correction was employed to avoid multiple-comparison errors. The significance level was set to 0.05, and only the corrected p-values are reported in this study. We conducted the statistical analyses with SPSS software of version 22.0.

3. Results

Figure 6 shows an example of the recognition results of subject S1. We used the t-SNE algorithm⁶⁶ to visualize the feature distributions and transformed the high-dimensional features into a two-dimensional feature space. Before the adaption process, the boundaries of MU features among gesture patterns were unclear, which could lead to numerous classification errors. After the adaption process, the separability became relatively clear with a few patterns still mixed up. Compared with the conventional EMG methods, the co-decision of activated MUs can effectively avoid possible recognition errors (as shown for F2 and F3 in Fig. 6).

Figure 7 shows the accuracy of MU classification between the MU-based baseline method and the proposed method. The adaption process significantly improved the recognition accuracy (baseline: ($60.69\pm 4.7$)%; the proposed method: ($83.53\pm 2.63$)%) ($p<0.05$). Figure 8 shows the comparison results of the proposed MU-based method and state-of-the-art EMG methods. OT-ST outperforms all the other comparison methods. The average accuracies of the proposed method were ($93.94\pm 1.54$)%, which was significantly higher than that of SGAS ($88.43\pm 2.50$)%, LPMM ($90.33\pm 1.47$)% and OT-ST ($91.33\pm 1.79$)% ($p<0.05$). The recognition accuracies of different methods for each finger movement are also shown in Table 1 for all movement patterns.

Figure 9 shows the accuracy changes with the adaption process as data from a new participant were incrementally fed into the ST models after testing with representative participants (S1, S6, and S11). The horizontal axis represents the number of sample batches, and the vertical axis shows the current classification accuracy. As the batch number of samples gradually increased, the recognition accuracy simultaneously rose to a high level and then remained stable with minor fluctuations. This was also the case for all tested users in this study. This shows that the adaption process has a positive effect on the classification performance.

We calculated the average time delay over all time windows of testing samples. Our method required $0.135\pm 0.024$s for each time window and was always less than 0.2s (the increment of the time window). The results validated the real-time performance of the proposed method.^{16,17,18,19,22}

4. Discussion

An online method for cross-subject finger movement pattern recognition incorporating neural decoding approach is proposed in this study. Unlike conventional methods, our method is the first to employ UDA algorithms leveraging microscopic neural drive information to improve the robustness of a myoelectric control system. With the adaption process, the model effectively classified all the activated MUs. The FWD strategy also enhanced the performance of pattern classification. Our proposed method provides a brand-new sight to solve robustness problem of myoelectric control system from fine-grained microscopic neural drive information.

In the past 10 years, the field of myoelectric pattern recognition has been extensively investigated,^{41,42,51,52,53} which primarily relied on the powerful data capability of deep learning networks based on the global features of SEMG signals. However, this is just an oversimplified description of the physiological characteristics of SEMG signals and rich neural drive information has been neglected. With the development of SEMG decomposition, it is possible to mine microscopic neural drive information from SEMG signals to detect the movement intentions more accurately. It was only in the past few years that the ”ultimate feature” at microscopic MU level has been explicitly employed to achieve dexterous and intuitive myoelectric control.⁴⁵ The results showed that the proposed method significantly outperformed the conventional EMG methods, which can be attributed to the ability of the SEMG decomposition to mine more fine-grained information. The disturbance caused by the nonstationarity of SEMG signals can be effectively eliminated through the MU FWD voting. Although current decomposition methods cannot provide a full absolute view of the MU activities, we have attempted to find physiological explanations to overcome the cross-user problem of myoelectric control from all available neural drive information. The results demonstrate the effectiveness of decoding fine-grained microscopic neural drive information in a myoelectric control system.

Compared with the MU-based baseline method, there was a significant improvement in the accuracy of MU classification by the proposed method, as shown in Fig. 7. In addition, Fig. 6 shows that the boundaries among gesture patterns were relatively clear for correct recognition after adaption. This can be explained by the capability of the UDA in accommodating cross-user conditions.^51,52,53 Previous studies demonstrated this capability using global features,^51,52,53 and our study proved that it can be applied to microscopic features as well. To fully utilize the neural drive information contained in MU activities, an algorithm was designed for feature extraction from multi-channel MUAPTs. The multi-channel MUAPTs serve as the basic components of SEMG signals and can also be considered as a series of “multi-channel SEMG signals” with infinite signal-to-noise-ratio. Relevant studies have mainly focused on the MU discharge information,^42,43,44 and ignored the rich waveform and spatial information. Our study can highlight the potential of integrating feature extraction from MUAPTs into a myoelectric control system. In addition, we have compared the OT-ST framework used in this study with other EMG methods (LPMM⁵¹ and SGAS⁵²). The results showed that the OT-ST method was more accurate among these UDA-based methods. This can be explained by the transfer learning effectiveness of the student–teacher model and the pseudo label optimization of the OT algorithm, which has been discussed in detail in our previous work.⁵³

Although the UDA can enhance the MU classification accuracy, the feature distributions of some patterns (such as F2 and F3 in Fig. 6) still remained intertwined, resulting in a general MU classification accuracy of approximately 80%. The physiology of the co-activation between the fingers can elucidate the results. It is evident that there is not a straightforward one-to-one correspondence between MU activities and finger movements, so it is necessary to use the MU FWD voting strategy to characterize this complex many-to-many relationship. Although the MU classification accuracy was not very high, the correct movement classification result of a single time window can be obtained by considering all the activated MUs, thus avoiding the influence of some misclassified MUs. Aside from the MU voting strategy in this study, some researches on myoelectric control have attempted to refine MUs based on specific contraction tasks to mitigate the impact of co-activation on the performance.^67,68 This may achieve good performance in certain conditions, but much neural drive information is discarded. Full and comprehensive use of neural drive information is always one of the most important necessities for establishing an ideal neural myoelectric interface.¹⁸

To implement the proposed method in real time, we employed the online PFP method¹⁸ to extract MUSTs in real time. The results of time delay illustrated that our method meet the demands for processing in real time. The acceleration of data processing primarily relies on the transfer of MU separation vectors.^35,36,37,38 As a result, a trial of data from the testing subject was used to calculate the separation vectors for subsequent online decomposition process. This procedure can be viewed as an initialization step for functions in smart devices, such as fingerprint recognition. The real-time performance of our method illustrated the feasibility of its application in a practical myoelectric control system.

However, there are some limitations to this work. First, the size and increment of the time window were a little large, which was mainly due to the requirement of SEMG decomposition. Sufficient time for SEMG signals can help to ensure accurate decomposition and waveform estimation.⁶⁹ Future work will be done to help optimize this aspect and enhance the practicality of our method. Second, the calculation of the separation vectors can be optimized. To comprehensively obtain MU separation vectors, we intentionally chose one trial to perform this operation. However, this experimental protocol is still convenient in a practical neural interface. Third, additional hand gestures, such as closing and opening the hand, could also be investigated with our method to improve its generalization. Future research will focus on these limitations.

5. Conclusion

The cross-user gesture recognition is crucial for facilitating the widespread and practical use of myoelectric control system. Existing studies mainly relied on the global features of SEMG signals and ignored the rich microscopic neural drive information. With this consideration, this study is the first attempt to improve the cross-user robustness of myoelectric control by incorporating neural decoding approach with UDA algorithms. The microscopic features from MU activities were extracted through online SEMG decomposition technique based on the PFP algorithm and then they were fed into an OT-ST framework to adapt the model from the source domain to the target domain for precise MU classification. In addition, an MU FWD voting strategy was designed to establish a many-to-many correspondence to explain the possible physiological relationship of MU activities and finger movements. The results showed that the proposed method significantly outperformed conventional EMG methods based on global features. This method offers a new perspective to address robustness problems in myoelectric control systems based on microscopic neural information and could have potential applications in neural interfaces, biomedical engineering, and prosthesis control.

However, our method still relies on a relatively long window to precisely decompose SEMG signals. In practical applications, this may lead to challenges such as satisfying real-time requirements. In addition, the calculation of MU separation vectors and the experimental protocols also need to be optimized. To address these problems, our future work will focus on developing an efficient SEMG decomposition framework and extending the SEMG dataset.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant No. 62271464 and in part by the Anhui Provincial Key Research and Development Program under Grant 2022k07020002.

Abbreviation List

ORCID

Haowen Zhao  https://orcid.org/0009-0008-7712-6417

Yunfei Liu  https://orcid.org/0009-0003-3309-7291

Xinhui Li  https://orcid.org/0009-0006-5522-0037

Xiang Chen  https://orcid.org/0000-0001-8259-4815

Xu Zhang  https://orcid.org/0000-0002-1533-4340