Narrow Results

In colleges and universities, resources for ideological and political education are available in various forms, increasing the complexity and difficulty of resource classification. To address this challenge, we proposed an adaptive classification method based on deep learning. We used DSNet-based key frame summary model to process ideological and political education video resources, obtained key frame sequences and input them into the image feature extraction network with ResNet34 residual network and SE module to generate the video resource feature map. We input college ideological and political education text resources into the Word2Vec-based word vector model, convert words into word vectors, use TF-IDF and CTF-IDF to obtain the vector representation of text resources and input them into the ABLCNN-based text feature extraction network to generate the feature map of text resources. We use a multimodal fusion network to integrate the characteristics of textual and video materials for the classification of ideological and political educational content. The empirical findings show that the F1-Score metric peaks when 32 key frames are used and this approach effectively classifies ideological and political educational resources in higher education with a comprehensive classification precision of 98.87%.

Crowdsourced testing for mobile applications is widespread. Efficiently and swiftly inspecting enormous test reports is a crucial challenge in crowdsourced testing. Most existing methods focus on finding different bugs earlier, i.e. the diversity of bugs, while ignoring bug severity. We present a multi-objective prioritization method for crowdsourced test reports that takes into account the diversity and severity of bugs. In the beginning, we ranked the test reports according to severity, then clustered the test reports by fusing textual and image features, and finally used the clustering results to adjust the initial ranking to ensure diversity. To validate our method, we conducted experiments using an industrial crowdsourced test report dataset comprised of six mobile application projects. The results demonstrate that our method can find more different high-severity bugs earlier than existing methods.

Rapid human motion pose tracking has extensive applications in fields such as motion capture, intelligent monitoring, sports training, and physical health management. It can provide accurate data support, enhance safety monitoring, optimize training outcomes, and promote physical health. Traditional human pose tracking methods predominantly rely on either sensors or images for tracking, which often results in issues like low tracking accuracy and slow tracking speed. To address these problems, a rapid human motion pose tracking method based on improved deep reinforcement learning and multimodal fusion is proposed. First, this paper designs an overall architecture for rapid human motion pose tracking and utilizes a combination of monocular vision and sensors to extract and collect human motion data. Second, it constructs a complementary filter-based multimodal data fusion method to merge the multimodal data and extract the fused features. Finally, a multi-level attention network is employed to enhance the deep reinforcement learning network, using the fused features as input for training to achieve rapid human motion pose tracking. The results show that the proposed method can achieve efficient and stable human motion pose tracking in complex scenes, with a tracking accuracy of up to 85% and a shortest tracking time of 72ms, which has practical application value.

Cognitive processes induced by the specific task are underpinned by intrinsic anatomical structures with functional neural activation patterns. However, current covariance network analysis still pays much attention to brain morphologies or baseline activity due to the lack of an effective method for capturing the structural–functional covarying during tasks. Here, a multimodal covariance network (MCN) construction method was proposed to identify inter-regional covariations of the structural skeleton and functional activities by simultaneous magnetic resonance imaging and electroencephalogram (EEG). Results from two independent cohorts confirmed that MCNs could capture cognition-specific hierarchical modules in joint comprehensive multimodal features well, especially when time-resolved EEG was further integrated. The quantitative evaluation further demonstrates significantly larger modularity of MCN integrating fine-grained features from EEG. The application to the discovery cohort identified prominent modular covarying across the default mode and salience networks at rest, while the visual oddball task was accomplished by synchronous structural–functional cooperation within networks associated with attention control and working memory updating. Strikingly, the results of an external validation cohort showed a different covariant pattern corresponding to decision-specific cognitive modules. Overall, the results suggested that multimodal covariance analysis provides a reliable definition of multistate neural cognitive networks, further discloses modular-specific structural and functional co-variation.

The problem of human activity recognition (HAR) has been increasingly attracting the efforts of the research community, having several applications. It consists of recognizing human motion and/or behavior within a given image or a video sequence, using as input raw sensor measurements. In this paper, a multimodal approach addressing the task of video-based HAR is proposed. It is based on 3D visual data that are collected using an RGB + depth camera, resulting to both raw video and 3D skeletal sequences. These data are transformed into six different 2D image representations; four of them are in the spectral domain, another is a pseudo-colored image. The aforementioned representations are based on skeletal data. The last representation is a “dynamic” image which is actually an artificially created image that summarizes RGB data of the whole video sequence, in a visually comprehensible way. In order to classify a given activity video, first, all the aforementioned 2D images are extracted and then six trained convolutional neural networks are used so as to extract visual features. The latter are fused so as to form a single feature vector and are fed into a support vector machine for classification into human activities. For evaluation purposes, a challenging motion activity recognition dataset is used, while single-view, cross-view and cross-subject experiments are performed. Moreover, the proposed approach is compared to three other state-of-the-art methods, demonstrating superior performance in most experiments.

Tourism recommendation systems have tended to become popular in recent years. Due to the fact that tourism content is generally with the format of multimodal information, existing research works mostly ignored the fusion of various feature types. To deal with this issue, this paper resorts to multimodal fusion of semantic analysis and image comprehension, and proposes a novel deep learning-based recommender system for tourism routes. First, semantic analysis under tourism route search is conducted, in order to complete destination selection and process selection. Then, image comprehension of overall tourism route planning is conducted by establishing an end-to-end object recognition model. Finally, the previous two parts of characteristics are fused together to formulate an integrated recommender system with multimodal sensing ability. This thought is expected to bring a stronger ability for tourism route discovery. Empirically, operational efficiency and stability analysis are carried out on real-world data to evaluate the performance of the proposal. The experimental results show that it can achieve significant improvement in tourism route recommendation, can accurately capture user preferences, and can provide travel suggestions that meet user requirements.

Facial expression recognition is an interesting research direction of pattern recognition and computer vision. It has been increasingly used in artificial intelligence, human–computer interaction and security monitoring. In recent years, Convolution Neural Network (CNN) as a deep learning technique and multiple classifier combination method has been applied to gain accurate results in classifying face expressions. In this paper, we propose a multimodal classification approach based on a local texture descriptor representation and a combination of CNN to recognize facial expression. Initially, in order to reduce the influence of redundant information, the preprocessing stage is performed using face detection, face image cropping and texture descriptors of Local Binary Pattern (LBP), Local Gradient Code (LGC), Local Directional Pattern (LDP) and Gradient Direction Pattern (GDP) calculation. Second, we construct a cascade CNN architecture using the multimodal data of each descriptor (CNN_LBP, CNN_LGC, CNN_GDP and CNN_LDP) to extract facial features and classify emotions. Finally, we apply aggregation techniques (sum and product rule) for each modality to combine the four multimodal outputs and thus obtain the final decision of our system. Experimental results using CK$+$ and JAFFE database show that the proposed multimodal classification system achieves superior recognition performance compared to the existing studies with classification accuracy of 97, 93% and 94, 45%, respectively.

Technology of brain–computer interface (BCI) provides a new way of communication and control without language or physical action. Brain signal tracking and positioning is the basis of BCI research, while brain modeling affects the treatment analysis of (EEG) and functional magnetic resonance imaging (fMRI) directly. This paper proposes human ellipsoid brain modeling method. Then, we use non-parametric spectral estimation method of time–frequency analysis to deal with simulation and real EEG of epilepsy patients, which utilizes both the high spatial and the high time resolution to improve the doctor’s diagnostic efficiency.