Narrow Results

Human action recognition based on depth video sequence is an important research direction in the field of computer vision. The present study proposed a classification framework based on hierarchical multi-view to resolve depth video sequence-based action recognition. Herein, considering the distinguishing feature of 3D human action space, we project the 3D human action image to three coordinate planes, so that the 3D depth image is converted to three 2D images, and then feed them to three subnets, respectively. With the increase of the number of layers, the representations of subnets are hierarchically fused to be the inputs of next layers. The final representations of the depth video sequence are fed into a single layer perceptron, and the final result is decided by the time accumulated through the output of the perceptron. We compare with other methods on two publicly available datasets, and we also verify the proposed method through the human action database acquired by our Kinect system. Our experimental results demonstrate that our model has high computational efficiency and achieves the performance of state-of-the-art method.

Musculoskeletal disorders (MSDs) are associated with sitting postures. The assessment and prevention of risk factors for workplace exposure are indispensable aspects of reducing the occurrence of MSDs. This paper proposes an ergonomic assessment method of risk factors for MSDs associated with sitting postures in the actual working conditions. A Kinect sensor with the RULA method was primarily used to collect the data and evaluate the relevant postures. The results obtained were compared with the evaluation results by a human expert. Additionally, we verified the capability and effectiveness of this method. A program system for human joint recognition and acquisition was implemented. The results indicated that the Kinect joint data is generally accurate and can adequately complete the RULA evaluation table. The results from the front and right-hand side obtained by the Kinect were consistent with the results of the expert evaluation, and no significant difference was observed between them ($p>0.05$). However, when the participants faced the Kinect, the sensor performed better, and the evaluation result was more accurate. A high consistency was observed between the evaluation results obtained from the front and the expert (proportion agreement $\mathrm{\text{index}}=0.65$, Cohen’s $\mathrm{\text{kappa}}=0.77$). Only a slight consistency was observed between the evaluation results obtained from the right-hand side and the expert (proportion agreement $\mathrm{\text{index}}=0.41$, Cohen’s $\mathrm{\text{kappa}}=0.08$). This research created a new ergonomic method for the risk assessment of MSDs associated with sitting postures. The combination of theory and practice is crucial in the risk assessment of sitting postures in workplaces.

Existing video communication systems, used in private business or video teleconference, show a part of the body of users on display only. This situation does not have realistic sensation because of showing a part of body on display and showing other users by 2D. This makes users feel communicating with the other users at a long distance without realistic sensation. Furthermore, although these existing communication systems have file transfer function such as sending and receiving file data, it does not use intuitive manipulation. It uses mouse or touching display only. In order to solve these problems, we propose 3D communication system supported by Kinect and head mounted display (HMD) to provide users communications with realistic sensation and intuitive manipulation. This system is able to show whole body of users on HMD as if they were in the same room by 3D reconstruction. It also enables users to transfer and share information by intuitive manipulation to augmented reality (AR) objects toward the other users in the future. The result of this paper is a system that extracts human body by using Kinect, reconstructs extracted human body on HMD, and also recognizes users' hands to be able to manipulate AR objects by a hand.

For motor learning tasks, different feedback methods to the user may impact the efficiency of motor training. Microsoft Kinect has shown great potential as an assistive device for home-based exercise training. This study introduces a Kinect-based skeleton-matching feedback system and compares its training outcomes with those of traditional mirror feedback in order to explore its applicability for use in home-based motor training. Forty healthy adults were randomly divided into an experimental group and a control group, and were instructed to perform a set of continuous shoulder motions using Kinect-based skeleton-matching feedback and traditional mirror feedback respectively. Fisher’s z-transformed Pearson’s correlation coefficient ($r_z$) and root-mean-square error (RMSE) were used to evaluate angular differences in shoulder movements between the subject’s actual motion and a reference motion for each group. Compared to the results under mirror feedback, r values with skeleton-matching feedback were significantly improved for symmetric abduction (SA), asymmetric abduction (ASA), and symmetric flexion/extension (SFE) ($P<0.05$), and RMSE values were significantly decreased for symmetric flexion/extension (SFE) ($P<0.001$). Therefore, the Kinect-based skeleton-matching feedback system boasts greater training efficiency for shoulder motor learning and has great potential as an assistive device for home-based rehabilitation exercise training.

Gesture recognition plays an important role in human–computer interaction. However, most existing methods are complex and time-consuming, which limit the use of gesture recognition in real-time environments. In this paper, we propose a static gesture recognition system that combines depth information and skeleton data to classify gestures. Through feature fusion, hand digit gestures of 0–9 can be recognized accurately and efficiently. According to the experimental results, the proposed gesture recognition system is effective and robust, which is invariant to complex background, illumination changes, reversal, structural distortion, rotation, etc. We have tested the system both online and offline which proved that our system is satisfactory to real-time requirements, and therefore it can be applied to gesture recognition in real-world human–computer interaction systems.

Facial expression recognition has been widely used in human computer interaction (HCI) systems. Over the years, researchers have proposed different feature descriptors, implemented different classification methods, and carried out a number of experiments on various datasets for automatic facial expression recognition. However, most of them used 2D static images or 2D video sequences for the recognition task. The main limitations of 2D-based analysis are problems associated with variations in pose and illumination, which reduce the recognition accuracy. Therefore, an alternative way is to incorporate depth information acquired by 3D sensor, because it is invariant in both pose and illumination. In this paper, we present a two-stream convolutional neural network (CNN)-based facial expression recognition system and test it on our own RGB-D facial expression dataset collected by Microsoft Kinect for XBOX in unspontaneous scenarios since Kinect is an inexpensive and portable device to capture both RGB and depth information. Our fully annotated dataset includes seven expressions (i.e., neutral, sadness, disgust, fear, happiness, anger, and surprise) for 15 subjects (9 males and 6 females) aged from 20 to 25. The two individual CNNs are identical in architecture but do not share parameters. To combine the detection results produced by these two CNNs, we propose the late fusion approach. The experimental results demonstrate that the proposed two-stream network using RGB-D images is superior to that of using only RGB images or depth images.

We never stop finding better ways to communicate with machines. To interact with computers we tried several ways, from punched tape and tape reader to QWERTY keyboards and command lines, from graphic user interface and mouse to multi-touch screens. The way we communicate with computers or devices are getting more direct and easier. In this paper, we give gesture mouse simulation in human–computer interface based on 3 Gear Systems using two Kinect sensors. The Kinect sensor is the perfect device to achieve dynamic gesture tracking and pose recognition. We hope the 3 Gear Systems can work as a mouse, to be more specific, use gestures to do click, double click and scroll. We use Coordinate Converting Matrix and Kalman Filter to reduce the shaking caused by errors and makes the interface create a better user experience. Finally the future of human–computer interface is discussed.

This paper improves the algorithm of point cloud filtering and registration in 3D modeling, aiming for smaller sampling error and shorter processing time of point cloud data. Based on collaborative sampling among several Kinect devices, we analyze the deficiency of current filtering algorithm, and use a novel method of point cloud filtering. Meanwhile, we use Fast Point Feature Histogram (FPFH) algorithm for feature extraction and point cloud registration. Compared with the aligning process using Point Feature Histograms (PFH), it only takes 9min when the number of points is about 500,000, shortening the aligning time by 47.1%. To measure the accuracy of the registration, we propose an algorithm which calculates the average distance of the corresponding coincident parts of two point clouds, and we improve the accuracy to an average distance of 0.7mm. In the surface reconstruction section, we adopt Ball Pivoting algorithm for surface reconstruction, obtaining image with higher accuracy in a shorter time.

Biometric identification of the human face is a pervasive subject which deals with a wide range of disciplines such as image processing, computer vision, pattern recognition, artificial intelligence, and cognitive psychology. Extracting key face points for developing software and commercial devices of face surgery analysis is one of the most challenging fields in computer image and vision processing. Many studies have developed a variety of techniques to extract facial features from color and gray images. In recent years, using depth information has opened up new approaches to researchers in the field of image processing. Hence, in this study, a statistical method is proposed to extract key nose points from color-depth images (RGB-D) of the face front view. In this study, the Microsoft Kinect sensor is used to produce the face RGB-D images. To assess the capability of the proposed method, this algorithm is applied to 20 RGB-D face images from the database collected in the ICT lab of Sahand University of Technology and promising results are achieved for extracting key points of the face. The results of this study indicated that using the available information in two different color-depth bands could make key points of the face more easily accessible and bring better results and we can conclude that the proposed algorithm provided a promising outcome for extracting the positions of key points.

In order to alleviate the phenomenon of healthcare resource shortage, this paper proposes a gait detection method using Kinect camera for post-operation patients with lower extremity injuries to carry out rehabilitation training by themselves at home. By using in depth data captured from the Kinect camera, we extract the 3D coordinates of the skeleton joints and use the space vector method to calculate the knee bending angle and stride angle as gait features. Based on these two features and imitated data, we developed a gait detection classifier using SVM. In the experiment, 500 abnormal gaits and 500 normal gaits are used as the training data and 100 gaits from experimental subjects are used as the test data. The experimental results revealed a sensitivity value of 88% and an algorithm accuracy of 91%, demonstrating that the proposed method has good classification capability and high detection rates for abnormal gait. This assists patients with lower extremities which cannot be bent freely to learn about the rehabilitation training conditions.

To solve the problem of complex operations in the current smart home system, acquire a simple control method and increase feeling experience for users, the gesture recognition method based on Kinect was researched and integrated into the Human-Computer Interaction (HCI) system in smart home. Users can customize the gestures to realize the intelligent control of household equipment. The utilized template matching gesture recognition method is accomplished based on the Dynamic Time Warping (DTW) algorithm. The results of actual gesture recognition experiments show that the gesture recognition based on DTW is feasible and effective. The best identification distance is in the range of 2 - 2.5 m in front of Kinect and the highest recognition accuracy is reached on 96%.

A full-body recognition method of human motions is proposed in this paper. It allows human to control the humanoid robot to imitate his motions with full-body language. In proposed Human-Robot Interaction system (HRI), real human manipulation motions are acquired by Microsoft’s Kinect sensor. To improve the efficiency of the motion recognition, the particle filter algorithm is proposed. In terms of the challenge for a humanoid robot to imitate full-body human motions, two different strategies are used in upper body and lower body respectively. The upper-body system includes a series of angle calculations while a Finite-State Machine (FSM) is introduced in lower-body recognition process. The corresponding simulations and experiments are implemented with both software and a self-made humanoid robot. The results indicate the proposed method is stable and efficient.

This paper presents the architecture of a localization system for affordable service robots. The system utilizes a RGB-D sensor for visual global localization based on artificial landmarks. The unobtrusive and cheap landmarks are based on QR-codes. We demonstrate how to recognize printed QR-codes over a wide range of viewing ranges and angles, and compute the robot pose uncertainty related to the landmark measurement. The localization method is integrated in the Robot Operating System and demonstrated in a home-like environment.

To overcome the instability of 3D body scanning by manual operation on Kinect sensor, such as skipping frame, missing data, and time consuming, an automatic 3D scanning device using a single Kinect sensor is designed in terms of the KinectFusion algorithm. The control system of computer - Arduino controller - motor drive - stepper motor - power supply has been employed to realize the flap motion and the circular motion of Kinect. The efficiency of the scanning has been improved by automatic control. The optimized scanning parameters have been determined via various scanning tests. Compared to the hand-held scanning, the experimental results indicate that the time cost can be reduced, and the quality of point cloud obtained can be improved significantly.

Most of the human body tracking algorithm is not applicable because of false recognition of position of human body. Depth tracking algorithm is designed to solve this problem according to the depth information of human motion based on Kinect sensor. By the analysis of Kinect depth map information, we distinguish the contour of the human body, extract the foreground area and reconstruct the depth histograms of the target area. Based on the depth histogram analysis, the position of joints of human body is obtained The experimental results show that the algorithm to extract the target feature is accurate and robust with avoiding the illumination changes and background under similar circumstances. The research is helpful for doctors to judge the status of fracture patients after operation.

For the problem that when doing image segment, the object extraction is susceptible to interference. Based on the saliency map and the model of human retinal receptive field, this paper presents a new method forimage segment. By calculating the map of gazing points gotten using the Kinect, based on human retinal receptive field model, the comprehensive fusion map is obtained, so as to determine the limited area of the segment. Several experiments are done by the method put forward in this paper and the experimental results show that the algorithm is able to solve the problem of susceptible to interference in target segment, while ensuring robustness and meeting the real-time and accuracy needed. And the algorithm in this paper not only improves the speed and robustness on the promise of guaranteeing the segmenting accuracy, but also gives attribution to the field of image segmenting.