International Journal of Pattern Recognition and Artificial Intelligence

Gatherings of thousands to millions of people frequently occur for an enormous variety of educational, social, sporting, and political events, and automated counting of these high-density crowds is useful for safety, management, and measuring significance of an event. In this work, we show that the regularly accepted labeling scheme of crowd density maps for training deep neural networks may not be the most effective one. We propose an alternative inverse k-nearest neighbor (i$k$NN) map mechanism that, even when used directly in existing state-of-the-art network structures, shows superior performance. We also provide new network architecture mechanisms that we demonstrate in our own MUD-i$k$NN network architecture, which uses multi-scale drop-in replacement upsampling via transposed convolutions to take full advantage of the provided i$k$NN labeling. This upsampling combined with the i$k$NN maps further improves crowd counting accuracy. We further analyze several variations of the i$k$NN labeling mechanism, which apply transformations on the $k$NN measure before generating the map, in order to consider the impact of camera perspective views, image resolutions, and the changing rates of the mapping functions. To alleviate the effects of crowd density changes in each image, we also introduce an attenuation mechanism in the i$k$NN mapping. Experimentally, we show that inverse square root $k$NN map variation (iR$k$NN) provides the best performance. Discussions are provided on computational complexity, label resolutions, the gains in mapping and upsampling, and details of critical cases such as various crowd counts, uneven crowd densities, and crowd occlusions.

Event cameras are biologically-inspired sensors that gather the temporal evolution of the scene. They capture pixel-wise brightness variations and output a corresponding stream of asynchronous events. Despite having multiple advantages with respect to conventional cameras, their use is limited due to the scarce compatibility of asynchronous event streams with traditional data processing and vision algorithms. In this regard, we present a framework that synthesizes RGB frames from the output stream of an event camera and an initial or a periodic set of color key-frames. The deep learning-based frame synthesis framework consists of an adversarial image-to-image architecture and a recurrent module. Two public event-based datasets, DDD17 and MVSEC, are used to obtain qualitative and quantitative per-pixel and perceptual results. In addition, we converted into event frames two additional well-known datasets, namely Kitti and Cityscapes, in order to present semantic results, in terms of object detection and semantic segmentation accuracy. Extensive experimental evaluation confirms the quality and the capability of the proposed approach of synthesizing frame sequences from color key-frames and sequences of intermediate events.

With the recent technological advances, surveillance cameras became accessible to the general public and a huge amount of nonstructured data is being gathered. However, extracting value from this data is challenging, especially for tasks that involve human images, such as face recognition and person re-identification. Annotation of this kind of data is a challenging and expensive task. In this work, we propose a domain adaptation workflow to allow CNNs that were trained in one domain to be applied to another domain without the need for annotated target data. Our method uses AlignedReID++ as the baseline, trained using a Triplet loss with batch hard. Domain adaptation is done in an unsupervised manner by clustering unlabeled data to generate pseudo-labels in the target domain. Our results show that domain adaptation really improves the performance of the CNN when applied in the target domain.

Building good 3D maps is a challenging and expensive task, which requires high-quality sensors and careful, time-consuming scanning. We seek to reduce the cost of building good reconstructions by correcting views of existing low-quality ones in a post-hoc fashion using learnt priors over surfaces and appearance. We train a convolutional neural network model to predict the difference in inverse-depth from varying viewpoints of two meshes — one of low-quality that we wish to correct, and one of high-quality that we use as a reference. Our full model runs at 11.3Hz when aggregating four input views. In contrast to previous work, we pay attention to the problem of excessive smoothing in corrected meshes. We address this with a suitable network architecture, and introduce a loss-weighting mechanism that emphasizes edges in the prediction. Furthermore, smooth predictions result in geometrical inconsistencies. To deal with this issue, we present a loss function which penalizes re-projection differences that are not due to occlusions. Future applications of this work will incorporate semantic scene understanding in a multi-task learning setting. We explore the efficacy of the proposed system in terms of gross error correction and generalization capability by showing its performance in practice on a subset of the Kitti Odometry dataset, complete with a component-wise ablation study. We evaluate correctness and completeness measures of surface reconstruction across viewpoints and show that the proposed system is introspective in regions lacking sufficient high-quality supervision — indeed, models trained with geometric consistency loss create a lot more surface in areas that were not supervised, in one case filling in 67.97% or 8010m² of an unlabeled input region. Finally, we assess the practical applicability of our method at large-scale by experiments over the full scope of the Kitti Odometry dataset. Broadly, as a measure of effectiveness, our model reduces gross errors by 45.3–77.5%, up to five times more than previous work. We also assess the practical applicability of our method to 3D reconstruction at large scales and find that compared to the baseline our model shows better stability in correctness when improving completeness of surfaces, and is effective in reducing median total error by up to 21.8cm.

This paper describes a deep learning-based method for long-term video interpolation that generates intermediate frames between two music performance videos of a person playing a specific instrument. Recent advances in deep learning techniques have successfully generated realistic images with high-fidelity and high-resolution in short-term video interpolation. However, there is still room for improvement in long-term video interpolation due to lack of resolution and temporal consistency of the generated video. Particularly in music performance videos, the music and human performance motion need to be synchronized. We solved these problems by using human poses and music features essential for music performance in long-term video interpolation. By closely matching human poses with music and videos, it is possible to generate intermediate frames that synchronize with the music. Specifically, we obtain the human poses of the last frame of the first video and the first frame of the second video in the performance videos to be interpolated as key poses. Then, our encoder–decoder network estimates the human poses in the intermediate frames from the obtained key poses, with the music features as the condition. In order to construct an end-to-end network, we utilize a differentiable network that transforms the estimated human poses in vector form into the human pose in image form, such as human stick figures. Finally, a video-to-video synthesis network uses the stick figures to generate intermediate frames between two music performance videos. We found that the generated performance videos were of higher quality than the baseline method through quantitative experiments.

Text-guided image editing (TIE) seeks to manipulate images by the guidance of language. However, the existing TIE methods always overlook the target-irrelevant pixels and the editing may make the background discolored, distorted, or partially disappear. To overcome this problem, we propose a novel TIE method named FocusGAN, which edits the text-relevant pixels precisely as well as keeps the background invariant. Specifically, we build a network of two stages. In each stage, we first construct a channel-wise subject focusing attention to make the generator focus on the sub-region that best matches each word. Then, the word-level discriminator provides fine-grained feedback by correlating words with image areas, so that the generator can manipulate specific visual attributes without affecting the background. Last, we propose a background-keeping cyclic loss to further improve the invariance of the background and to encourage the edit of the subject that matches the given text. Experiments on CUB and Oxford datasets demonstrate that our approach can effectively keep the background invariant in manipulating images using natural language descriptions.

Remote sensing techniques have been developed over the past decades to acquire data without being in contact of the target object or data source. Their application on land-cover image segmentation has attracted significant attention during recent years. With the help of satellites, scientists and researchers can collect and store high-resolution image data that can be further be processed, segmented, and classified. However, these research results have not yet been synthesized to provide coherent guidance on the effect of variant land-cover segmentation processes. In this paper, we present a novel model that augments segmentation using smaller networks to segment individual classes. The combined network is trained on the same data but with the masks, combined and trained using categorical cross entropy. Experimental results show that the proposed method produces the highest mean IoU (Intersection of Union) as compared against several existing state-of-the-art models on the DeepGlobe dataset.

In the intelligently processing of the tongue image, one of the most important tasks is to accurately segment the tongue body from a whole tongue image, and the good quality of tongue body edge processing is of great significance for the relevant tongue feature extraction. To improve the performance of the segmentation model for tongue images, we propose an efficient tongue segmentation model based on U-Net. Three important studies are launched, including optimizing the model’s main network, innovating a new network to specially handle tongue edge cutting and proposing a weighted binary cross-entropy loss function. The purpose of optimizing the tongue image main segmentation network is to make the model recognize the foreground and background features for the tongue image as well as possible. A novel tongue edge segmentation network is used to focus on handling the tongue edge because the edge of the tongue contains a number of important information. Furthermore, the advantageous loss function proposed is to be adopted to enhance the pixel supervision corresponding to tongue images. Moreover, thanks to a lack of tongue image resources on Traditional Chinese Medicine (TCM), some special measures are adopted to augment training samples.

Various comparing experiments on two datasets were conducted to verify the performance of the segmentation model. The experimental results indicate that the loss rate of our model converges faster than the others. It is proved that our model has better stability and robustness of segmentation for tongue image from poor environment. The experimental results also indicate that our model outperforms the state-of-the-art ones in aspects of the two most important tongue image segmentation indexes: IoU and Dice. Moreover, experimental results on augmentation samples demonstrate our model have better performances.

Permanent-magnet direct-drive belt conveyors (PMDDBCs) rotate at high speed most of the time, resulting in a large number of invalid energy consumption. To realize the speed regulation of PMDDBC, it is necessary to clarify the relationship between the belt speed, coal quantity of the conveyor and total power of the system. Based on the BP neural network, this paper establishes the power consumption model of PMDDBC, which is related to coal quantity, belt speed and total power. Furthermore, an improved hybrid algorithm (GACO) that combines the advantages of genetic algorithm (GA) and ant colony optimization (ACO) is proposed to optimize the BP power consumption model. The GACO–BP power consumption model is obtained. The original power consumption model is compared with the GACO–BP power consumption model through experiments. Results demonstrate that the GACO–BP power consumption model reduces various prediction errors, while the optimization ability, prediction accuracy and convergence speed are significantly enhanced. It provides a reliable speed regulation basis for the permanent-magnet direct-drive belt conveyor system and also provides a theoretical reference for energy savings and consumption reduction in the coal industry.

A mobile community can be composed of multiple mobile devices through D2D (Device-to-Device) network. In many cases, these mobile devices cannot conveniently connect to the Internet, for various reasons. To overcome this obstacle, one solution is to let the mobile devices cooperate with each other through a D2D-enabled network, forming a mobile community that, as a whole, may be able to autonomously execute the tasks requested by its members. To maximize the overall benefits of mobile communities, this paper proposes a novel task allocation approach, EDTG (Energy-aware and Deadline-constrained Task allocation using Game theory). In mobile communities, energy consumption is responsible for the largest part of the cost. Energy management can lead to performance degradation and even be perceived as a bottleneck, while load balancing between devices can improve service performance and resource utilization to the largest extent. EDTG has considered both the inevitable performance constraints at each device and a method based on the connectivity of graph theory, in order to narrow down the search scope of optimal target mobile devices where requested tasks can be executed. The “Bargaining Game” method is designed and exploited to obtain the final task allocation solution. Final experimental results demonstrate that compared to existing approaches, EDTG ensures high-performance task execution and reaches the goal of maximizing the overall benefits to some extent, by achieving better energy savings and exploiting load balancing between devices.

E-commerce retailers face the challenge to assemble a large number of time-critical picking orders. Common parts-to-picker autonomous intelligent warehouses such as automated vehicle storage and retrieval system and robotic mobile fulfillment system are often a little ill-suited for these prerequisites. A mixed-robotic fulfillment system is a hybrid robot picking system based on multi-device collaboration. It is a fusion innovation of traditional automated vehicle storage and retrieval system and robotic mobile fulfillment system. This paper comprehensively considers the characteristics of the system and customer demand, through the construction of a queuing network model to evaluate the performance of the system. A series of problems such as order service time, throughput capacity, and vehicle quantity configuration are analyzed experimentally. The validity of the model is verified by a simulation model.

Designing efficient algorithms for the large-scale multi-objective problems (LSMOPs) is a very challenging problem currently. To tackle LSMOPs, we first design a new reference points selection strategy to enhance the diversity of algorithms and avoid the algorithm falling into local minima. The strategy selects not only a part of nondominated solutions with the largest crowding distance, but also a part of relatively uniformly distributed solutions as the reference points. In this way, much better diversity and convergence can be obtained. Second, we propose a direction-guided offspring generation strategy, where a type of potential directions is designed to generate the promising solutions which can balance the convergence and diversity of the obtained solutions and improve the effectiveness of the algorithm significantly. Based on the two proposed strategies, we propose a new effective algorithm for LSMOPs. Numerical experiments are executed on two widely used large-scale multi-objective benchmark problem sets with 200, 500 and 1000 decision variables and a comparison with five state-of-the-art algorithms is made. The experimental results show that our proposed algorithm is effective and can obtain significantly better solutions than the compared algorithms.

To improve the specification and applicability of performance indicators for in-vehicle subtask impact experiments, additional driving performance indicators were added to the international standard test environment, the lane-changing test (LCT). The comprehensive characteristics of multiple indicators were studied using the C4.5 decision tree. The paired memory test was selected as a subtask, and data on driving trajectory, speed, and steering angle were collected and analyzed under three task conditions (baseline, easy task, and difficult task). The results showed that the newly added indicators such as lane offset times, initial lane-changing distance, and correct lane-changing ratio could reflect the new driving characteristics from the aspects of the times of lateral deviation over threshold events, driver response, error proportion, etc. Subjects with different driving states were grouped into the lane-keeping indicator and lane-changing indicator. By using the C4.5 decision tree, classification accuracies of 85.2% and 91.3% were achieved, respectively, which indicated a high accuracy rate of task state discrimination. The overall performance of lane keeping and lane changing was different under the action of the subtasks, especially the lane-changing performance.