International Journal of Humanoid Robotics

Electromyography (EMG) has been widely accepted to interact with prosthetic hands, but still limited to using few channels for the control of few degrees of freedom. The use of more channels can improve the controllability, but it also increases system’s complexity and reduces its wearability. It is yet clear if optimizely placing the EMG channel could provide a feasible solution to this challenge. This study customized a genetic algorithm to optimize the number of channels and its position on the forearm in inter-day hand gesture recognition scenario. Our experimental results demonstrate that optimally selected 14 channels out of 16 can reach a peak inter-day hand gesture recognition accuracy at 72.3%, and optimally selecting 9 and 11 channels would reduce the performance by 3% and 10%. The cross-validation results also demonstrate that the optimally selected EMG channels from five subjects also work on the rest of the subjects, improving the accuracies by 3.09% and 4.5% in 9- and 11-channel combination, respectively. In sum, this study demonstrates the feasibility of channel reduction through genetic algorithm, and preliminary proves the significance of EMG channel optimization for human–machine interaction.

Deep learning (DL) has made tremendous contributions to image processing. Recently, the DL has also attracted attention in the specialized field of neural decoding from raw myoelectric signals (electromyograms, EMGs). However, to our knowledge, most existing methods require some measure of preprocessing of the raw EMGs. Moreover, research to date has not accounted for the variability in the signal during time sequences. In this paper, we propose a new convolutional neural network (CNN) structure that can directly process raw EMG signals for hand gesture classification. More specifically, we assess the effects of various window sizes and of two different EMG representations (time sequence and frequency spectra) on open EMG datasets. We found that the frequency spectra derived from raw EMGs is more suitable as the model input in the task of gesture classification. Meanwhile, the combination use of long window could improve the classification accuracy (CA) and the window of 1024 ms achieved the best results on two open datasets ($73.5\pm 5.5$% and $91.7\pm 2.5$%). Further, our model requires no feature extraction procedures and is comparable with the optimal combination of features and classifier used by the traditional methods in the performance of specific tasks.

Facial expression recognition has been widely used in human computer interaction (HCI) systems. Over the years, researchers have proposed different feature descriptors, implemented different classification methods, and carried out a number of experiments on various datasets for automatic facial expression recognition. However, most of them used 2D static images or 2D video sequences for the recognition task. The main limitations of 2D-based analysis are problems associated with variations in pose and illumination, which reduce the recognition accuracy. Therefore, an alternative way is to incorporate depth information acquired by 3D sensor, because it is invariant in both pose and illumination. In this paper, we present a two-stream convolutional neural network (CNN)-based facial expression recognition system and test it on our own RGB-D facial expression dataset collected by Microsoft Kinect for XBOX in unspontaneous scenarios since Kinect is an inexpensive and portable device to capture both RGB and depth information. Our fully annotated dataset includes seven expressions (i.e., neutral, sadness, disgust, fear, happiness, anger, and surprise) for 15 subjects (9 males and 6 females) aged from 20 to 25. The two individual CNNs are identical in architecture but do not share parameters. To combine the detection results produced by these two CNNs, we propose the late fusion approach. The experimental results demonstrate that the proposed two-stream network using RGB-D images is superior to that of using only RGB images or depth images.

This paper details an intelligent motion planning and control approach for a one-degree of freedom joint of a robotic arm that can be used to implement anthropomorphic robotic hands. This intelligent control method is based on bio-inspired electronic neural networks and contractile artificial muscles implemented with shape memory alloy (SMA) actuators. The spiking neural network (SNN) includes several excitatory neurons that naturally determine the contraction force of the actuators, and unevenly distributed inhibitory neurons that regulate the excitatory activity. To validate the proposed concept, the experiments highlight the motion planning and control of a single-joint robotic arm. The results show that the electronic neural network is able to intelligently activate motion and hold with high precision the mobile link to the target positions even if the arm is slightly loaded. These results are encouraging for the development of improved biologically plausible neural structures that are able to control simultaneously multiple muscles.

Localization of vessel Region of Interest (ROI) from medical images provides an interactive approach that can assist doctors in evaluating carotid artery diseases. Accurate vessel detection is a prerequisite for the following procedures, like wall segmentation, plaque identification and 3D reconstruction. Deep learning models such as CNN have been widely used in medical image processing, and achieve state-of-the-art performance. Faster R-CNN is one of the most representative and successful methods for object detection. Using outputs of feature maps in different layers has been proved to be a useful way to improve the detection performance, however, the common method is to ensemble outputs of different layers directly, and the special characteristic and different importance of each layer haven’t been considered. In this work, we introduce a new network named Attention Layer R-CNN(AL R-CNN) and use it for automatic carotid artery detection, in which we integrate a new module named Attention Layer Part (ALP) into a basic Faster R-CNN system for better assembling feature maps of different layers. Experimental results on carotid dataset show that our method surpasses other state-of-the-art object detection systems.

Human pose estimation is a fundamental but challenging task in computer vision. The estimation of human pose mainly depends on the global information of the keypoint type and the local information of the keypoint location. However, the consistency of the cascading process makes it difficult for each stacking network to form a differentiation and collaboration mechanism. In order to solve these problems, this paper introduces a new human pose estimation framework called Multi-Scale Collaborative (MSC) network. The pre-processing network forms feature maps of different sizes, and dispatches them to various locations of the stack network, with small-scale features reaching the front-end stacking network and large-scale features reaching the back-end stacking network. A new loss function is proposed for MSC network. Different keypoints have different weight coefficients of loss function at different scales, and the keypoint weight coefficients are dynamically adjusted from the top hourglass network to the bottom hourglass network. Experimental results show that the proposed method is competitive in MPII and LSP challenge leaderboard among the state-of-the-art methods.

Motion detection and object tracking play important roles in unsupervised human–machine interaction systems. Nevertheless, the human–machine interaction would become invalid when the system fails to detect the scene objects correctly due to occlusion and limited field of view. Thus, robust long-term tracking of scene objects is vital. In this paper, we present a 3D motion detection and long-term tracking system with simultaneous 3D reconstruction of dynamic objects. In order to achieve the high precision motion detection, an optimization framework with a novel motion pose estimation energy function is provided in the proposed method by which the 3D motion pose of each object can be estimated independently. We also develop an accurate object-tracking method which combines 2D visual information and depth. We incorporate a novel boundary-optimization segmentation based on 2D visual information and depth to improve the robustness of tracking significantly. Besides, we also introduce a new fusion and updating strategy in the 3D reconstruction process. This strategy brings higher robustness to 3D motion detection. Experiments results show that, for synthetic sequences, the root-mean-square error (RMSE) of our system is much smaller than Co-Fusion (CF); our system performs extremely well in 3D motion detection accuracy. In the case of occlusion or out-of-view on real scene data, CF will suffer the loss of tracking or object-label changing, by contrast, our system can always keep the robust tracking and maintain the correct labels for each dynamic object. Therefore, our system is robust to occlusion and out-of-view application scenarios.

Aggressive driving, such as frequent lane changes, endangers other persons or property but is challenging to be continuously tracked by existing traffic surveillance systems. In this paper, we use wrist-mounted wearables, such as a smartwatch to monitor the driver’s forearm acceleration and to detect lane changes. Because the forearm acceleration of lane changes can be significantly affected by traveling speed and steering angle, our system transforms the time-domain acceleration data to the frequency domain for clearly depicting the signal distribution over a range of frequencies. To further improve detection accuracy, we develop an adaptive algorithm which dynamically determines the target frequency band and adjusts the signal energy evaluation threshold based on current traveling speed. The algorithm will also examine the signal energy distribution over other frequencies besides the target frequency band to avoid false alarms when driving on road curves. We have evaluated our system in real driving environments, including both the low-speed local roads and high-speed expressways, and the results show that the system achieves high detection accuracy at low computational complexity.

Epilepsy is caused by sudden abnormal discharges of neurons in the brain. This paper constructs an automatic seizure detection system, which combines the predicting result of multi-domain feature with the predicting result of spike rate feature to detect the occurrence of epileptic seizures. After segmenting EEG data into 5s with 80% overlap epochs, the paper extracts time domain features, frequency domain features and hurst exponents (HE) from each epoch and these features are reduced by linear discriminant analysis (LDA) to be input parameters of the random forest (RF) classifier, which provides classification of the EEG epochs concerning the existence of seizures. In parallel, the paper extracts spikes from EEG data with morphological filter and calculates the spike rate to determine whether there is seizure. Then the results obtained by these two methods are merged as the final detection result. The paper shows that the accuracy (AC), sensitivity (SE), specificity (SP) and the false positive ratio based on event (FPRE) obtained by hybrid method are 98.94%, 76.60%, 98.99% and 2.43 times/h, respectively. Finally, the paper applies the seizure detection method to do seizure warning and recording to help the family member to take care of the patients and the doctor to adjust the antiepileptic drugs (AEDs).

The corticomuscular coupling (CMC) characterization between the motor cortex and muscles during motion control is a valid biomarker of motor system function after stroke, which can improve clinical decision-making. However, traditional CMC analysis is mainly based on the coherence method that can’t determine the coupling direction, whereas Granger Causality (GC) is limited in identifying linear cause–effect relationship. In this paper, a time-frequency domain copula-based GC (copula-GC) method is proposed to assess CMC characteristic. The 32-channel electroencephalogram (EEG) signals over brain scalp and electromyography (EMG) signals from upper limb were recorded during controlling and maintaining steady-state force output for five stroke patients and five healthy controls. Then, the time-frequency copula-GC analysis was applied to evaluate the CMC strength in both directions. Experimental results show that the CMC strength of descending direction is greater than that of ascending direction in the time domain for healthy controls. With the increase of grip strength, the bi-directional CMC strength has an increasing trend. Meanwhile, the bi-directional CMC strength of right hand is larger than that of left hand. In addition, the bi-directional CMC strength of stroke patients is lower than that of healthy controls. In the frequency domain, the strongest CMC is observed at the beta frequency band. Additionally, the CMC strength of descending direction is slightly larger than that of ascending direction in healthy controls, while the CMC strength of descending direction is lower than that of ascending direction in stroke patients. We suggest that the proposed time-frequency domain analysis approach based on copula-GC can effectively detect complex functional coupling between cortical oscillations and muscle activities, and provide a potential quantitative analysis measure for motion control and rehabilitation evaluation.

Due to the extremely weak intensity of the biomagnetic field and the serious interference from the environmental magnetic field, the detection of the biomagnetic field becomes such challenging work. After analyzing the deficiencies in the current biomagnetic field sensors, this paper proposes and realizes a high-sensitivity magnetic field sensor, based on the giant magneto-impedance (GMI) effect. Taking advantage of the miniaturized magnetic probe, the multistage multiple amplification and the multiband interference suppression, our sensor mainly makes three achievements: the pT level magnetic resolution, the ability to detect the muscle magnetic field without the magnetic shielding and the resistibility to a small-range wobbling in the state of working, which makes it possible to detect the biomagnetic field by wearable sensors under natural conditions.

Surface electromyography (sEMG) signals have been widely used in human–machine interaction, providing more nature control expedience for external devices. However, due to the instability of sEMG, it is hard to extract consistent sEMG patterns for motion recognition. This paper proposes a dual-flow network to extract the temporal-spatial feature of sEMG for gesture recognition. The proposed network model uses convolutional neural network (CNN) and long short-term memory methods (LSTM) to, respectively, extract the spatial feature and temporal feature of sEMG, simultaneously. These features extracted by CNN and LSTM are merged into temporal-spatial feature to form an end-to-end network. A dataset was constructed for testing the performance of the network. In this database, the average recognition accuracy by using our dual-flow model reached 78.31%, which was improved by 6.69% compared to the baseline CNN (71.67%). In addition, NinaPro DB1 is also used to evaluate the proposed methods, receiving 1.86% higher recognition accuracy than the baseline CNN classifier. It is believed that the proposed dual-flow network owns the merit in extracting stable sEMG feature for gesture recognition, and can be further applied into practical applications.

Crowd feature perception is an essential step for us to understand the crowd behavior. However, as the individuals present not only the sociality but also the randomness, there remain great challenges to extract the sociality of the individual directly. In this paper, we propose a crowd feature perception algorithm based on a sparse linear model (SLM). It builds the statistical characterization of the sociality by assuming a priori distribution of the SLM. First, we calculate the optical flow to extract the motion information of the crowd. Second, we input the video motion features to the sparse coding and generate the SLM. The super-Gaussian prior distributions in SLMs build the statistical characterization of the sociality. In addition, we combine the infinite Hidden Markov Model (iHMM) statistic model to determine whether the detected event is an abnormal event. We validate our method on UMN dataset and simulate dataset for abnormal detection, and the experiments show that this algorithm generates promising result compared with other state-of-art methods.

The channel numbers and electrode layouts are usually determined empirically that would reduce robustness when acquiring surface electromyography (EMG) signals for prosthetic hand systems. It is necessary to study how they can be exploited effectively for a more accurate extraction. In response to the problem, an experiment is designed that establishes the relationship between sEMG signals and forearm muscles based on signal-to-noise ratio (SNR). The SNR of sEMG signals in different sampling channels can be calculated and compared, and then the potential contribution of each channel during different hand motions will be evaluated comprehensively. The active muscle regions can be obtained from the established relationship that is a useful reference for feature extraction. Finally, the relations between the computational cost, channel numbers and electrode layouts are explored. The findings of this paper support the idea that the accuracy of pattern recognition will not be affected when reducing the redundant electrodes.