International Journal of Pattern Recognition and Artificial Intelligence

Sign language recognition (SLR) is a useful tool for the deaf-mute to communicate with the outside world. Although many SLR methods have been proposed and have demonstrated good performance, continuous SLR (CSLR) is still challenging. Meanwhile, due to the heavy occlusions and closely interacting motions, there is a higher requirement for the real-time efficiency of CSLR. Therefore, the performance of CSLR needs further improvement. The highlights include: (1) to overcome these challenges, this paper proposes a novel video-based CSLR framework. This framework consists of three components: an OpenPose-based skeleton stream extraction module, a RGB stream extraction module, and a combination module of the BiLSTM network and the conditional hidden Markov model (CHMM) for CSLR. (2) A new residual network with Squeeze-and-Excitation blocks (SEResNet50) for video sequence feature extraction. (3) This paper combines the SEResNet50 module with the BiLSTM network to extract the feature information from video streams with different modalities. To evaluate the effectiveness of our proposed framework, experiments are conducted on two CSL datasets. The experimental results indicate that our method is superior to the methods in the literature.

Aiming at the poor convergence of the current sketch image classification and unable to meet the growing network needs, a classification and model method of convolution feature in sketch image based on deep learning is proposed. Based on the classification principle of deep learning, a classification experiment was carried out on the semantic features of sketch works through the analysis of convolution neural network, convolution feature model, convolution and sketch extraction boundary. The experimental results show that the proposed convolution classification and recognition method is better than the traditional classification method and has higher accuracy in dimensionality reduction and error rate detection than the traditional method. It can better meet the needs of network intelligent processing of sketch image feature classification.

Adversarial attacks can fool convolutional networks and make the systems vulnerable to fraud and deception. How to defend against malicious attacks is a critical challenge in practice. Adversarial attacks are often conducted by adding tiny perturbations on images to cause network misclassification. Noise reduction can defend the attacks; however, it is not suited for all the cases. Considering that different models have different tolerance abilities on adversarial attacks, we develop a novel detecting module to remove noise by adaptive process and detect adversarial attacks without modifying the models. Experimental results show that by comparing the classification results on adversarial samples of MNIST and two subclasses of ImageNet datasets, our models can successfully remove most of the noise and obtain detection accuracies of 97.71% and 92.96%, respectively. Furthermore, our adaptive module can be assembled into different networks to achieve detection accuracies of 70.83% and 71.96%, respectively, on the white-box adversarial attacks of ResNet18 and SCD01MLP images. The best accuracy of 62.5% is obtained for both networks when dealing with the black-box attacks.

Mathematical morphology intends to extract object features such as geometric and topological structures in digital images. Given a set of target images and original images, it is cumbersome and time-consuming to determine the suitable morphological operations and structuring elements. In this paper, we propose deep morphological neural networks, which include a nonlinear feature extraction layer to learn the structuring element correctly and an adaptive layer to select appropriate morphological operations automatically. We demonstrate the applications of object recognition, including hand-written digits, geometric shapes, traffic signs, and brain tumor. Experimental results show the higher computational efficiency and higher accuracy of our developed model as compared against existing convolutional neural network models.

Multi-modal dense video captioning is a task using multiple information to detect all meaningful events and generate a textual description for each event. The existing works mainly rely on single visual or dual audio-visual modals in dense video captioning, while completely ignoring the text modal (subtitle). The text modal has a similar data structure as the video captions, which provides immediate semantic information to the content description for a video. In this paper, we propose a novel framework, called Two-Stage Cross-Modal Encoding Transformer Network (TS-CMETN), to realize the multi-modal dense video captioning task by fusing multiple features, including audio, visual, and text. First, we design a two-stage feature fusion encoder that hierarchically achieves the intra- and inter-modal information interaction. Second, we propose an anchor-free temporal event proposal module, which efficiently generates event proposals at each time step without the complex anchor calculation. Extensive experiments on the ActivityNet Captions dataset show that our proposed framework achieves high performance. Moreover, our approach can adaptively handle cases of the missing text modal. Our code and data are available at https://github.com/xieyulai/TM-CMETN.

Pupil detection is an indispensable part of the process of eye-tracking. Due of the limitation of existing methods on pupil image quality, we propose a pupil detection method using vision transformer with a hybrid structure. We first extract the local features of the image with CNN, and then obtain the global dependence through the encoder of the transformer, to excavate more accurate information on pupil position. We trained and tested the proposed model on 10 600 images from three publicly available datasets and compared with other pupil detection models. The analysis of the outcomes demonstrated that the hybrid vision transformer was superior to these comparison approaches in terms of accuracy and robustness in locating the pupil position. It achieved a detection rate of more than 90% for pupils within a 5-pixel error in all evaluated datasets.

Three-dimensional (3D) human pose estimation aims to locate 3D keypoints of individuals from given input RGB images. For two-dimensional (2D) human pose estimation problems, majority methods inferring 2D poses are from 2D heatmaps. However, it is hard to extend this method to 3D poses inferring area which makes computational loads increase sharply. To address the above problem, we propose STM-CNN method to estimate reconstruction coefficient matrix to calculate the final 3D pose instead of estimating 3D heatmaps to decrease the computational loads. First, STM-CNN does a preprocessing procedure to calculate a set of shape and weight bases. Second, STM-CNN infers a 2D matrix called reconstruction coefficient from the STM-CNN architecture. Third, STM-CNN utilizes the preprocessing shape and weight bases and estimated reconstruction coefficient matrix to calculate the final 3D pose. Meanwhile, STM-CNN method achieves better performances compared with the state-of-the-art methods on Human3.6M.

To fully pay attention to identity-sensitive feature information and utilize the correlations of inter-attributes and attributes-body parts, this paper proposes a person re-identification (re-ID) method based on the construction of graph convolutional network (GCN) with crucial attribute feature and body parts. First, it establishes the multiscale context-aware network (MSCAN) using dilated convolution with different expansion ratios, which can learn multiscale context information and obtain diversified global features. Subsequently, the human parsing model is utilized to extract the body part features. According to the attribute importance degree, the paper constructs low-dimensional GCN integrating the vital attributes and body parts of person descriptions to obtain discriminative local features. Finally, based on attribute prediction, it reduces the range of the images to be matched with discriminating possible objects from query images, thereby simplifying retrieval process. The experimental results demonstrate that the novel designed method can effectively improve person re-ID performance and achieve competitive evaluation results on typical public testing datasets.

Monitoring corn crop growth status is of great significance to crop production, breeding, and seed production. The Unmanned Aerial Vehicles’ (UAVs) technology makes it possible to use computer vision technology to identify corn growth stage intelligently. A model customized for corn growth status monitoring based on a dense convolutional neural network (CM-CNN) was proposed, including a two-way dense module and a new activation function ELU. The two-way dense module enlarges the receptive field, while the ELU alleviates gradient disappearance and speeds up learning in deep neural networks. Dense architecture concatenates all the previous layer features to enhance feature reuse. The proposed CM-CNN performs well in classifying corn growth stages. Experimental results show that CM-CNN is a state-of-the-art method, with an accuracy of its relevant data up to 99.3%. Compared with other CNN models, viz. AlexNet, ZFNet, VGG, InceptionV3, Xception and ResNet, fewer parameters are in CM-CNN.

The dynamic behaviors for fractional-order Cohen–Grossberg neural networks with time-varying delays (FCGNND) are studied in this paper. By introducing the Mittag-Leffler (ML) function, based on properties of fractional calculus, the differential mean-value theorem and Arzela–Ascoli theorem, we give some sufficient theorems to determine the boundedness, global Mittag-Leffler stability (GMLS) and global asymptotical $\omega$-periodicity (GAP) for FCGNND. Finally, a numerical example is given to verify the effectiveness of the theorems.

Functional Distributed Constraint Optimization Problems (F-DCOPs) are a constraint processing framework for continuous variables in the multi-agent system. Particle swarm optimization-based F-DCOP (PFD) is a population-based algorithm to solve F-DCOP collaboratively. Although it can significantly reduce the computational overhead and memory requirements, its solution depends on the decision of root agent in the Breadth First Search (BFS) pseudo-tree and it is easy to fall into local optimum. To solve the above problems, this paper designed an improved PFD algorithm with Local Decision named PFD-LD, which effectively reduces the dependence on root agent through local decision. In addition, a mutation operator is used to avoid falling into local optimum. It is proved that PFD-LD is an anytime algorithm and local decision can expand the search of the solution space. Finally, the extensive experiments based on four types of benchmark problems show that the proposed algorithm outperforms state-of-the-art F-DCOP solving algorithms.

Data clustering is the task of separating data samples into a set of clusters. $K$-means is a popular partitional clustering algorithm. However, it has a lot of weaknesses, including sensitivity to initialization and the ability to become stuck in local optima. Hence, nature-inspired optimization algorithms were applied to the clustering problem to overcome the limitations of the $K$-means algorithm. However, due to the high-dimensionality of a search space, the nature-inspired optimization algorithm suffers from local optima and poor convergence rates. To address the mentioned issues, this paper presents a simplex method-based bacterial colony optimization (SMBCO) algorithm. The simplex method is a stochastic variant approach that improves population diversity while increasing the algorithm’s local searching ability. The potential and effectiveness of the proposed SMBCO clustering algorithm are assessed using a variety of benchmark machine learning datasets and the generated groups were evaluated using different performance measures. When compared to several well-known nature-inspired algorithms, the experimental results reveal that the SMBCO model produces superior clustering efficiency and a faster convergence rate.

Model pruning is one of the main methods of deep neural network model compression. However, the existing model pruning methods are inefficient, there is still a lot of redundancy in the network, and the pruning has a great impact on the accuracy. In addition, the traditional pruning process usually needs to fine-tune the network to restore accuracy, and fine-tuning has a limited effect on accuracy recovery, so it is difficult to achieve a high level of accuracy. In this paper, we propose a new neural network pruning framework: the channel sparsity is realized by introducing a scale factor, and the sparse network is pruned by setting a global threshold, which greatly improves the efficiency of pruning. After pruning, this paper proposes a rewind method to restore the accuracy, that is, save the weight after training, and then reload it on the network for training to restore the accuracy. In addition, we also study the best rewind point of the three networks. The experimental results show that our method significantly reduces the number of parameters and FLOPs without affecting or even improving the accuracy, and the rewind method proposed by us achieves a better accuracy recovery effect than fine-tuning. At the same time, we find that the epoch with the highest accuracy is the best rewind point, and the accuracy is the highest after saving its corresponding weight and retraining the model.

Nowadays, increasing automatic guided vehicles (AGVs) are being introduced into the production line. The path planning and scheduling of multiple AGVs are complex tasks that are closely related to application scenarios. In this study, a robot’s working area was divided to minimize the collision probability. Improved ant colony optimization (ACO) and improved probabilistic road map (PRM) algorithms were used for path planning to enhance the transportation efficiency. The priorities of AGVs were specified. The collision criteria were determined, and the responses for different collision types were provided.