International Journal of Semantic Computing

Over the last decades, there has been growing interest in research in multiple and interdisciplinary fields of human-AI computing. In particular, approaches integrating human’s perspective and design with reinforcement learning (RL) have received more attention. However, the current research on RL may need to consider its enhancement from human-inspired approaches further. In this work, we focus on enabling a meta-reinforcement learning (meta-RL) agent to achieve adaptation and generalization, according to modeling Markov Decision Processes (MDP) using Bayesian knowledge and analysis. By introducing a novel framework called human-inspired meta-RL (HMRL), we incorporate the agent performing resilient actions to leverage the dynamic dense reward based on the knowledge and prediction of a Bayesian analysis. The proposed framework can make the agent learn generalization and prevent the agent from failing catastrophically. The experimental results show that our approach helps the agent reduce computational costs with learning adaptation. In addition to the system design, we have also extended further algorithmic improvement based on learning within a deep Q-network (DQN) implementations for more complicated future tasks, which compared replay buffers to possibly enhance the optimization process. Finally, we conclude and anticipate that integrating human-inspired meta-RL can enable learning more formulations relating to robustness and scalability, leading to promising directions and more complex AI goals in the future.

There is tremendous value in the ability to predict stock market trends and outcomes. The public sentiment surrounding a stock is unquestionably a vital factor contributing to the rise or fall of a stock price. This paper aims to detail how data from public sentiment can be integrated into traditional stock analyses and how these analyses can then be used to make predictions of stock price trends. Headlines from seven news publications and conversations from Yahoo! Finance’s conversations forum were processed by the Valence Aware Dictionary and sEntiment Reasoner (VADER) natural language processing package to determine numerical polarities that represent a positive, negative, or neutral public sentiment around a stock ticker. The resulting polarities were paired with popular stock-table metrics to create a dataset for a Logistic Regression machine learning model. The model was trained on approximately 4400 major stocks to determine a binary “Buy” (1) or “Not Buy” (0) recommendation for each stock. We present approaches for a variety of machine learning models – Logistic Regression, Random Forest Classifiers, and Extreme Gradient Boosting models and do a statistical comparison on the balanced accuracies of each machine learning model. This paper aims to detail our understanding of how one can leverage the public sentiment of a stock to gauge its future value.

Despite the great advances in the field of image classification, the association of ideal approaches that can bring improved results, considering different datasets, is still an open challenge. In this work, a novel approach is presented, based on a combination of compared strategies: feature extraction for early fusion; rankings based on manifold learning for late fusion; and feature augmentation applied in a long short-term memory (LSTM) algorithm. The proposed method aims to investigate the effect of feature fusion (early fusion) and ranking fusion (late fusion) in the final results of image classification. The experimental results showed that the proposed strategies improved the accuracy of results in different tested datasets (such as CIFAR10, Stanford Dogs, Linnaeus 5, Flowers 102, and Flowers 17) using a fusion of features from three convolutional neural networks (CNNs) (ResNet152, VGG16, and DPN92) and its respective generated rankings. The results indicated significant improvements and showed the potential of the approach proposed for image classification.

Video streaming systems aim to provide high-quality video adapted to clients’ device and network conditions. For this purpose, adaptive streaming architectures encode video content at a variety of quality levels, organized in a bitrate ladder. However, compressing a video into multiple streams is resource-intensive, which may become especially problematic in live streaming applications with real-time demands. Therefore, this paper proposes a novel solution for fast bitrate ladder creation, and provides the requirements for implementation in the H.266/VVC standard. More specifically, the proposed method creates new intermediate combined streams by injecting the lowest temporal layers of a higher-quality augmentation stream in a lower-quality base stream. Since the lowest layers are used as reference by the remaining layers, this procedure indirectly increases the quality of the frames in those untouched remaining layers as well. Although altering the reference frames may cause some drift-error artifacts, we demonstrate that, on average, the quality of the intermediate streams are higher than the base stream. Additionally, we demonstrate that injecting more layers brings both the quality and bitrate closer to that of the augmentation stream. The disadvantage of the combined streams is that their quality fluctuates more than the quality of the source streams, and that they are compressed less efficiently, comparable to going from a slower to fast or faster preset in the VVenC encoder. Most importantly, their main advantage is that they were generated at no significant additional computational complexity. In this way, the proposed method is of great benefit when generating a bitrate ladder of video streams under constrained computational resources. The code has been made open source and is available on https://github.com/IDLabMedia/NALUProcessing.

In the rapidly evolving landscape of digital platforms, the need for optimizing media representations to cater to various aspect ratios is palpable. In this paper, we pioneer an approach that utilizes object detection, scene detection, outlier detection, and interpolation for smart cropping. Using soccer as a case study, our primary goal is to capture the frame salience using object (player and ball) detection and tracking using AI models. To improve the object detection and tracking, we rely on scene understanding and explore various outlier detection and interpolation techniques. Our pipeline, called SmartCrop, is efficient, and supports various configurations for object tracking, interpolation, and outlier detection to find the best point-of-interest to be used as the cropping center of the video frame. An objective evaluation of the performance of individual pipeline components has validated our proposed architecture. Moreover, a crowdsourced subjective user study, assessing the alternative approaches for cropping from 16:9 to 1:1 and 9:16 aspect ratios, confirms that our proposed approach increases the end-user quality of experience.

The reading experience on a mobile device can be influenced by many external elements that affect the viewing conditions. The user holding position, operating style, and the light conditions of the surroundings can all play a crucial factor in the legibility of the text. Furthermore, the user’s eye conditions may be an extra factor to consider when determining how to improve the reading experience. This study presents a context-aware application that adjusts the text font size based on the distance between the user and the device. The users select their preferred font size at certain distances in the calibration phase, accounting for individual preferences or their eye conditions, and the application uses the device’s front camera to calculate the distance in real time and adjust the font size accordingly. We present implementation details and challenges faced during development and how they were addressed. A two-part experiment ($n$ = 34) revealed that the application scored an excellent usability score while no differences were found between users with and without eye conditions, showing that both user groups find it equally usable. Additionally, initial results suggest that the visual landmarks used showed a promising start at addressing issues related to our approach.

Advancements in artificial intelligence have significantly increased the use of images and videos in machine analysis algorithms, predominantly neural networks. However, the traditional methods of compressing, storing and transmitting media have been optimized for human viewers rather than machines. Current research in coding images and videos for machine analysis has evolved in two distinct paths. The first is characterized by End-to-End (E2E) learned codes, which show promising results in image coding but have yet to match the performance of leading Conventional Video Codecs (CVC) and suffer from a lack of interoperability. The second path optimizes CVC, such as the Versatile Video Coding (VVC) standard, for machine-oriented reconstruction. Although CVC-based approaches enjoy widespread hardware and software compatibility and interoperability, they often fall short in machine task performance, especially at lower bitrates. This paper proposes a novel hybrid codec for machines named NN-VVC, which combines the advantages of an E2E-learned image codec and a CVC to achieve high performance in both image and video coding for machines. Our experiments show that the proposed system achieved up to − 43.20% and − 26.8% Bjøntegaard Delta rate reduction over VVC for image and video data, respectively, when evaluated on multiple different datasets and machine vision tasks according to the common test conditions designed by the VCM study group in MPEG standardization activities. Furthermore, to improve reconstruction quality, we introduce a human-focused branch into our codec, enhancing the visual appeal of reconstructions intended for human supervision of the machine-oriented main branch.

This work addresses the importance of incorporating multi-scale information in image representation by proposing a novel approach utilizing hierarchical segmentation and graph neural networks (GNNs). The proposed model, named Hierarchical Image Graph with Scale Importance (HIGSI), leverages hierarchical segmentation to construct graphs that capture relationships between nodes across different scales. This multi-scale representation simultaneously captures intricate details and global context, leading to a richer understanding of image structure than traditional methods. Additionally, a novel Region Graph Readout (RGR) function is introduced to assess the significance of each scale within the graph representation. By combining this multi-scale representation and the RGR function, HIGSI achieves competitive performance on image classification tasks, using smaller graphs or having fewer parameters than existing methods. This work also presents a comparative study with another hierarchical approach and an assessment of HIGSI’s components to investigate its decision-making process and its components’ contribution to the overall performance.