International Journal of Semantic Computing

With the growth of crowd phenomena in the real world, crowd scene understanding is becoming an important task in anomaly detection and public security. Visual ambiguities and occlusions, high density, low mobility, and scene semantics, however, make this problem a great challenge. In this paper, we propose an end-to-end deep architecture, convolutional nonlinear differential recurrent neural networks (CNDRNNs), for crowd scene understanding. CNDRNNs consist of GoogleNet Inception V3 convolutional neural networks (CNNs) and nonlinear differential recurrent neural networks (RNNs). Different from traditional non-end-to-end solutions which separate the steps of feature extraction and parameter learning, CNDRNN utilizes a unified deep model to optimize the parameters of CNN and RNN hand in hand. It thus has the potential of generating a more harmonious model. The proposed architecture takes sequential raw image data as input, and does not rely on tracklet or trajectory detection. It thus has clear advantages over the traditional flow-based and trajectory-based methods, especially in challenging crowd scenarios of high density and low mobility. Taking advantage of CNN and RNN, CNDRNN can effectively analyze the crowd semantics. Specifically, CNN is good at modeling the semantic crowd scene information. On the other hand, nonlinear differential RNN models the motion information. The individual and increasing orders of derivative of states (DoS) in differential RNN can progressively build up the ability of the long short-term memory (LSTM) gates to detect different levels of salient dynamical patterns in deeper stacked layers modeling higher orders of DoS. Lastly, existing LSTM-based crowd scene solutions explore deep temporal information and are claimed to be “deep in time.” Our proposed method CNDRNN, however, models the spatial and temporal information in a unified architecture and achieves “deep in space and time.” Extensive performance studies on the Violent-Flows, CUHK Crowd, and NUS-HGA datasets show that the proposed technique significantly outperforms state-of-the-art methods.

We introduce KaraMIR, a musical project dedicated to karaoke song analysis. Within KaraMIR, we define Kara1k, a dataset composed of 1000 cover songs provided by Recisio Karafun application, and the corresponding 1000 songs by the original artists. Kara1k is mainly dedicated toward cover song identification and singing voice analysis. For both tasks, Kara1k offers novel approaches, as each cover song is a studio-recorded song with the same arrangement as the original recording, but with different singers and musicians. Essentia, harmony-analyser, Marsyas, Vamp plugins and YAAFE have been used to extract audio features for each track in Kara1k. We provide metadata such as the title, genre, original artist, year, International Standard Recording Code and the ground truths for the singer’s gender, backing vocals, duets, and lyrics’ language. KaraMIR project focuses on defining new problems and describing features and tools to solve them. We thus provide a comparison of traditional and new features for a cover song identification task using statistical methods, as well as the dynamic time warping method on chroma, MFCC, chords, keys, and chord distance features. A supporting experiment on the singer gender classification task is also proposed. The KaraMIR project website facilitates the continuous research.

The task of visual relationship recognition (VRR) is to recognize multiple objects and their relationships in an image. A fundamental difficulty of this task is class–number scalability, since the number of possible relationships we need to consider causes combinatorial explosion. Another difficulty of this task is modeling how to avoid outputting semantically redundant relationships. To overcome these challenges, this paper proposes a novel architecture with a recurrent neural network (RNN) and triplet unit (TU). The RNN allows our model to be optimized for outputting a sequence of relationships. By optimizing our model to a semantically diverse relationship sequence, we increase the variety in output relationships. At each step of the RNN, our TU enables the model to classify a relationship while achieving class–number scalability by decomposing a relationship into a subject–predicate–object (SPO) triplet. We evaluate our model on various datasets and compare the results to a baseline. These experimental results show our model’s superior recall and precision with fewer predictions compared to the baseline, even as it produces greater variety in relationships.

With the widespread availability of media technologies, such as real-time streaming, new Internet-of-Thing devices and smart phones, multimedia data are extensively increased and the big multimedia data rapidly spread over various social networks. This has created complexity and information overload for users to choose the suitable multimedia objects. Thus, different multimedia recommender systems have been emerging to help users find the useful multimedia objects that are possibly preferred by the user. However, the evaluation of these multimedia recommender systems is still in an ad-hoc stage. Given the distinct features of multimedia objects, the evaluation criteria adopted from the general recommender systems might not be effectively used to evaluate multimedia recommendations. In this paper, we therefore review and analyze the evaluation criteria that have been used in the previous multimedia recommender system papers. Based on the review, we propose a generalized evaluation framework to guide the researchers and practitioners to perform evaluations, especially user-centric evaluations, for multimedia recommender systems.

This paper describes the use of semantically annotated data for the expression of sensors and actuators with their properties and capabilities. For this purpose, a plug and play mechanism is presented in order to exchange self-descriptions between several hardware devices and then use the established information for the execution of capabilities. For the combination of different capabilities distributed on multiple hardware devices an architecture is presented to link abstract capabilities. These abstract capabilities are initialized through concrete capabilities of the discovered hardware devices by the plug and play mechanism.

A method for enhancing the communication between instructor and student is to automate the procedure of replying to the student’s questions. In order to achieve that, techniques of textual case-based reasoning are used, benefiting both the instructor and the student. The automation of this procedure significantly decreases the workload and involvement required by the tutor. In addition to that, the time required for the students to receive an answer is greatly decreased. In this paper we present such a system, based on pre-existing knowledge, in order to provide answers quickly, for the questions asked by the students.

Graph-based Online Analytical Processing (GOLAP) extends Online Analytical Processing (OLAP) to address graph-based problems that involve object attributes. Based on graph data, GOLAP can answer user queries related to combinatorial optimization, structural analytics, and influence analytics. Besides, since a GOLAP system is an online interactive system that requires fast response time, the execution time for graph-problem queries is essentially critical. Thus, how to speed up the execution time of specific graph problems becomes a challenge in GOLAP. In this paper, we show several methods to speed up the running time, including graph data reduction and approximation. In this paper, we survey classes of graph-based queries, challenges for GOLAP, and solutions that GOLAP provides.