Free Access

Optimized Feature Selection Approach with Elicit Conditional Generative Adversarial Network Based Class Balancing Approach for Multimodal Sentiment Analysis in Car Reviews

Professor and Head of Academics, Exafluence Education, (A EdTech unit of Exafluence Inc., USA), @ Sri Venkateswara University, Tirupati, Andhra Pradesh-517502, India

E-mail Address: sriramankothuri@gmail.com

Corresponding author.

Search for more papers by this author

and

N. R. Rajalakshmi

Professor, Department of Computer Science and Engineering, School of Computing, Vel Tech Rangarajan Dr. Sagunthala R&D Institute of Science and Technology, Avadi, Chennai, Tamil Nadu-600062, India

E-mail Address: drnrrajalakshmi@veltech.edu.in

Search for more papers by this author

https://doi.org/10.1142/S0218488525500035Cited by:0 (Source: Crossref)

Abstract

Multimodal Sentiment Analysis (MSA) is a growing area of emotional computing that involves analyzing data from three different modalities. Gathering data from Multimodal Sentiment analysis in Car Reviews (MuSe-CaR) is challenging due to data imbalance across modalities. To address this, an effective data augmentation approach is proposed by combining dynamic synthetic minority oversampling with a multimodal elicitation conditional generative adversarial network for emotion recognition using audio, text, and visual data. The balanced data is then fed into a granular elastic-net regression with a hybrid feature selection method based on dandelion fick’s law optimization to analyze sentiments. The selected features are input into a multilabel wavelet convolutional neural network to classify emotion states accurately. The proposed approach, implemented in python, outperforms existing methods in terms of trustworthiness (0.695), arousal (0.723), and valence (0.6245) on the car review dataset. Additionally, the feature selection method achieves high accuracy (99.65%), recall (99.45%), and precision (99.66%). This demonstrates the effectiveness of the proposed MSA approach, even with three modalities of data.

Keywords:

1. Introduction

With the increasing wealth of information available on the internet, individuals can enhance their product choices and lifestyles based on textual details.¹ Businesses, recognizing the importance of understanding customer opinions, employ sentiment analysis through social media monitoring, brand tracking, and emotions found in emails and comments.² Sentiment analysis helps to deduce the emotional tone underlying text to comprehend user opinions.² However, analyzing sentiment in large datasets, such as user reviews, poses challenges due to variations in sequence length and textual order. Deep learning approaches are introduced in text classification and question answering to address these challenges.^3,4

Machine learning and deep learning techniques were applied to MSA, even in domains such as analyzing sentiments from coronavirus-related tweets.⁵ Innovative approaches, like Massive Open Online Courses (MOOC), have utilized sentiment classification methods organized by ensemble learning paradigms.⁶ The integration of deep learning approaches serves as a robust baseline model for feature extraction in sentiment analysis.^7,39,40 Various sentiment analysis methods use lexicon-based approaches to identify the orientation of text documents, while machine learning-based methods leverage labeled datasets to train models.^8,9

In the context of MSA, polarity detection is often considered the most relevant information.¹⁰ Classification tasks such as subjective classification, opinion extraction, word sentiment classification, and document sentiment classification are employed to identify classes in datasets.¹¹ The domain of sentiment analysis has witnessed growing interest in applications such as visual summaries and trend analysis.¹² Furthermore, hyperparameter optimization strategies play a crucial role in minimizing generalization errors in machine learning approaches.^13,14

Unlike single model sentiment analysis,^33,34 MSA poses several challenges and opportunities. Early approaches relied on handcrafted features, limiting sentiment abstraction and yielding suboptimal results. Explaining multimodal models is complex due to the need to relate model performance to diverse input data. The heterogeneity and dimensionality of human behaviors hinder feature interpretation and understanding of model decisions. Additionally, research on compact, human-friendly data summaries is scarce. Interpreting inter-modal interactions is non-trivial, despite their importance. For instance, discerning positive sentiment from neutral voice and facial cues remains challenging, highlighting the unique nature of MSA. Therefore, a new MSA approach is developed to differentiate emotions using different modalities such as audio, video, and text.

Motivation

In the digital age, user-generated content on online platforms has created a rich repository of multimodal data, including text, audio, and video. This diversity presents both an opportunity and a challenge for sentiment analysis, especially in understanding and interpreting human emotions comprehensively. MSA emerges as a critical field of emotional computing research, leveraging varied data to gain deeper insights into user sentiments. MSA integrates and analyzes multiple data modalities, providing an accurate understanding of sentiments compared to unimodal approaches. For instance, text alone might convey a different emotional tone when not considered alongside corresponding vocal intonations or facial expressions present in audio and video data. By combining these modalities, MSA offers a holistic view of sentiments, leading to more robust and reliable analysis outcomes. The application of MSA is particularly impactful in car review analysis, where user feedback encompasses various forms of expression. Car reviews often include detailed text descriptions, vocal commentary, and visual demonstrations, making them an ideal candidate for multimodal analysis. Understanding the sentiments behind these reviews provides automotive companies with valuable insights into customer preferences, and overall satisfaction. This aids in product improvement, marketing strategies, and customer engagement, ultimately enhancing the consumer experience and driving business success.

Contributions

Social media information is a valuable resource for sentiment analysis applications, aiding in the identification of user emotions to solve various problems. This work contributes to the field by emphasizing texture classification through spectral analysis.

This research work utilizes the Granular Elastic-Net Regression (GENR) model to enhance texture classification and improve the accuracy and robustness of MSA, especially in scenarios with intricate textual patterns.
The proposed Multilabel Wavelet Convolutional Neural Network (MWCNN) offers a comprehensive analysis by considering both low and high-frequency components in textual data, improving sensitivity to a wider range of features compared to traditional CNNs.
Integration of Dynamic Synthetic Minority Over-sampling TEchnique (DSMOTE) provides a sophisticated mechanism to address biases in dataset sampling, ensuring a more representative dataset compared to traditional methods.
Also, GENR is introduced to refine the regularization process and improve model adaptability, overcoming sparsity limitations inherent in Elastic Net Regression (ENR) for more precise sentiment analysis tasks.

This research work is organized as follows: An overview of several existing sentiment analyses with the MuSe-CaR dataset is provided in Section 2. The developed model is briefly explained in Section 3 with the steps of data acquisition, balancing class samples, feature selection, and classification. Section 4 gives a detailed explanation of the experimental result. Finally, the whole research work is concluded in Section 5.

2. Literature Survey

Exploring emotion recognition and sentiment analysis through multimodal data, the literature survey provides diverse methodologies. These range from extensive annotation and fusion techniques to unique feature extraction approaches. Various research works were previously carried out which were based on the MSA. Some of them are reviewed and listed in Table 1,

**Table 1. Comparison of existing methods.**
Authors	Methods	Pros	Cons
Stappen et al.¹⁵	Multi head Attention Network (MAN)	Continuous prediction capability of multitasking	Limited in the multimodal data analysis
Sun et al.¹⁶	Self-attention based Long Short-Term Memory (LSTM) with Recurrent Neural Network (RNN)	The performance is improved by continuous prediction	Limited to explore multitask learning and advanced fusion models
Stappen et al.¹⁷	Rater Aligned Annotation Weighting (RAAW)	It offered a cohesive approach to developing regression	Limited to reduced dimension
Baird et al.¹⁸	Lexical knowledge-based extraction approach	Low computational power and improved the linguistic baseline	Lacked to combine high-level features from different modalities using unsupervised methods
Vlasenko et al.¹⁹	Ensemble based classifier	Classification performance was improved by Natural Language Processing based approaches	Lacked gives an improved method by combining three modalities
Schuller et al.²⁰	Support VectorRegression (SVR)	Consistently more robust prediction for video likes	It never accurately predicts hate likes and fake news
Cambria et al.²¹	LSTM-RNN	High possible transparency with a large set of features	Limited to combine modalities at an earlier stage
Jiang et al.²²	Hybrid temporal modal	The best prediction result was achieved by multimodal features	Lacked to get better results with combined arousal and valence
Yadav et al.²³	An effective rater ensemble model	It takes all information about human emotions without losing any subjective information	More advanced architectures are needed to improve the performance (Accuracy: 77.89%) of the model
Padminivalli et al.²⁴	Spatial-temporal deep neural network	Overfitting is avoided by introducing cross-validation during testing and training	Struggled with recognizing neutral emotions (98.3 %)
Liu et al.³⁰	Modality Translation-based MSA model (MTMSA)	Minimizing the complexity of the MSA model	The accuracy of the suggested model sharply decreases as the missing rates increase (67.29% to 47.5%)
Chandrasekaran et al.³¹	Long short-term memory eXtreme Gradient Boosting (LXGB)	The suggested model excels in MSA, accurately classifying emotions across diverse data types	The model struggled with predicting neutral emotions (92.9%)
Wang et al.³²	Cross-Correlation in Dual-Attention (CCDA)	Better performance in MSA	Increased computational complexity

Stappen et al.¹⁵ have presented an extensive annotation process in covering all the emotions and selects the videos of having changing shots and dynamic backgrounds. The intensity class of trustworthiness is predicted based on audio and visual behaviors. Sun et al.¹⁶ have presented a human emotion/sentiment analysis with videos, audio, and text to automate sentiment analysis in many areas. The relevant features are extracted by using low-level and hybrid deep learning approaches to explore the robust feature extraction. The modal performance is again enhanced by introducing a fusion technique for arousal and valence.

Stappen et al.¹⁷ have presented an annotation toolkit to relate the various types of fusion techniques in continuous annotation. The relevant configuration parameter was pointed out in a series of manners to improve the capability of the tool kit. Another MSA modal was developed by Baird et al.¹⁸ for exploring sub-symbolic representation from sentic information into the emotional information provided from videos. It also provided the usefulness of high features from text and audio. With these better learning to predict valance and arousal was obtained.

Automatic emotion recognition is presented by Vlasenko et al.¹⁹ with the turn-level prediction of emotions using valence and arousal dimensions. Different approaches were investigated to fuse the text and audio features and various structures of deep learning were explored the cross-dependencies. Schuller et al.²⁰ have presented a first approach to feature extraction and selection without using audio video and text. There is no YouTube video prediction against time series features. The different choices between interpretability of accuracy were chosen for the prediction method.

Cambria et al.²¹ have presented a challenging sentiment analysis approach with a large set of features by using open source software. Then it described a detailed description of multimodal feature extraction with applied preprocessing and alignment for the baseline modelling. The best prediction outcomes were obtained using multimodal features. Jiang et al.²² have explained the mapping of continuous dimensional emotions to discrete classes. It predicted emotion classes based on audio and video about user reviews. For the multimodal features, feature fusion was the best for segmenting data.

Yadev et al.²³ have presented an emotional recognition with audio and video data. It also explored the fusion of ensemble prediction with several techniques. The main motive of this modal is to achieve the whole information from raw annotation to predicted emotions. Padminivalli et al.²⁴ have presented an audio-visual based block that is analyzed with the temporal convolutional network. Along, with these block leader followers’ attentive fusion block after that, the cross-modality fusion is achieved to get noise removed network.

Liu et al.³⁰ suggested an MTMSA robust to uncertain missing modalities. The model translates visual and audio data to text and fuses them into Missing Joint Features (MJFs). A transformer encoder, supervised by a pre-trained model, encodes MJFs to approximate complete modalities. The transformer decoder learns intermodal dependencies, facilitating sentiment classification. However, the suggested model’s accuracy sharply decreases with increased missing rates, impacting intermodal feature projection and affecting visual and auditory feature space projection onto text.

Chandrasekaran et al.³¹ utilized the hybrid LXGB technique, which combined LSTM and eXtreme Gradient Boosting (XGBoost) classifiers for MSA. The suggested model addressed emotional understanding across image, textual, and audio data, showcasing effectiveness in capturing different sentiments.

Wang et al.³² developed the CCDA model and utilized dual attention mechanisms to obtain inter and intramodal dynamics efficiently. The model incorporates a cross-correlation loss to acquire attention correlation and utilizes relevant coefficients for effective feature integration. However, these computations, relying on matrix multiplication, increase model complexity and involve redundant information.

The disadvantages in existing methods are considered to improve the proposed system by introducing separate hybrid methods for data balancing and feature selection. The developed scheme is aimed at attaining a better outcome with the dimension of arousal, valence, and trustworthiness.

Problem statement

Many approaches were proposed for analyzing MSA. However, the existing approaches are constrained by the lack of combining modalities, which may result in low result in prediction. Also, the existing method does not provide the outcome with all three emotional states such as arousal, valence, and trustworthiness. Since most of the sentiment analyses are data driven, the capability of machine learning methods is limited for training data. Looking at the existing MSA, it seems that all the people express positive opinions as there is a lack of negative opinions.^{16,17,18,19,20,21,22,23,24}

3. Proposed Methodology for MSA

For the users generating videos, audio, and text on online websites, it is important to understand human emotions and sentiments to analyze the reviews of a certain product. Figure 1 shows the workflow of this research work,

MSA is a rapidly growing field of emotional computing research where reviews are gathered from the MuSe-CaR dataset.
These reviews are in three different modalities such as audio, video, and text which has some challenges due to imbalance data, especially when it involves multimodal data from various modalities which is addressed by an effective data augmentation approach of combined DSMOTE with multimodal Elicit Conditional Generative Adversarial Network (ECGAN)
Then the balanced data is given to the GENR with Hybrid Dandlion Fick’s Law Optimization (HDFLO) based feature selection method.
The selected features are given to MWCNN for accurate classification of sentiment states as valence, arousal, and trustworthiness.

Fig. 1. The workflow of the proposed system.

3.1. Multimodal data collection

Data is sourced from MuSe-CaR, a large car review dataset with multimodal data, featuring videos with varied face angles, backgrounds and occlusions, audios with high noise, and text containing colloquialisms. Given these challenges, sentiment analysis faces imperfect classification accuracy. To address this, an imbalanced data augmentation approach is applied to achieve balanced datasets.

3.2. Data augmentation method for imbalanced data using combined DSMOTE with ECGAN

The proposed MSA system aims to categorize the data as three classes but the dataset is affected by an imbalanced class distribution problem which affects the performance of the classifier. ECGAN is combined with DSMOTE to get efficient balanced classes at the classifier without blind oversampling. Moreover, the existing models lack control over the locations where synthetic samples are produced in the data space, leading to two key issues: (1) noisy minority samples create more noisy samples during oversampling, and (2) noisy samples within the majority class area cause under-fitting due to class overlap. The proposed data augmentation method has different phases. Initially, data are separated by class labels. Next, minority class samples are categorized into border and noisy types to oversample borderline samples while ignoring noisy ones, avoiding misclassification. Finally, range-controlled oversampling prevents minority samples from being generated within majority class regions.

Initially, the border samples are oversampled to balance the dataset for generating new synthetic samples. Then, new synthetic samples ( $X_{n e w}$ ) are generated through randomly selected line segments between the minority sample ( $\hat{x_{i}}$ ) and neighbors ( $x_{i}$ ). The generated new samples are expressed as eqn. (1),

X n e w = x i + r * (̂ x i - x i) <math display="block" altimg="eq-00004.gif"><msub><mrow><mi>X</mi></mrow><mrow><mi>n</mi><mi>e</mi><mi>w</mi></mrow></msub><mo>=</mo><msub><mrow><mi>x</mi></mrow><mrow><mi>i</mi></mrow></msub><mo>+</mo><mi>r</mi><mo>*</mo><mo stretchy="false">(</mo><mover accent="true"><mrow><msub><mrow><mi>x</mi></mrow><mrow><mi>i</mi></mrow></msub></mrow><mo>̂</mo></mover><mo>-</mo><msub><mrow><mi>x</mi></mrow><mrow><mi>i</mi></mrow></msub><mo stretchy="false">)</mo></math> (1)

where, r is the random number in [0,1]. These samples are fed into ECGAN, which generates new samples with a distribution matching that of the dataset. ECGAN proves to be more suitable for multimodal translation tasks. It incorporates an additional input (

$X_{n e w}$ ) from DSMOTE, sampled from true data distributions based on the minimum amount of data, both from the minority and majority classes. The learned distributions are then used to generate synthetic images. The ECGAN architecture contains two networks: a generator G and a discriminator D, represented by differentiable functions. The discriminator D is trained by learning from the generator G. The noise vector z is taken by the generator network in a latent space and differentiable function where the fake image, audio, and text, are denoted as

$G (z) \to X$ . Simultaneously, the discriminator network distinguishes between raw data (labeled by 1) and artificially generated data (labeled by 0) by the generator network, expressed as

$D (X_{n e w}) \to [0, 1]$ . The objective function of ECGANs for the three data types (Video

$V (D, G)$ , audio

$A (D, G)$ and text

$T (D, G)$ ) is defined as Eq. (2),

where, the random noise vector and real images are represented as z and x, the generated and the real data distribution are denoted as

$p_{r} (x)$ and

$p_{z} (z)$ , E represents the expectation,

$X_{n e w}$ is the output of DSMOTE, G and D represents the representation of generator and discriminator. The generator goal is to minimize

$log (1 - D (G (z)))$ and the discriminator aims to maximize

$log (D (X_{n e w}))$ . The balanced dataset of audio, video, and text-based reviews with three classes is then given to select the relevant features.

3.3. Feature selection using GENR HDFLO

GENR is also applied to multimodal data, such as audio, video, and text by using recursive feature elimination to determine the appropriate number of features. In this context, each modality represents a different set of features, and the goal is to classify the most relevant features within each modality as well as potential interactions between modalities. The block diagram of the proposed feature selection (GENR HDFLO) method is shown in Fig. 2.

Fig. 2. The block diagram of GENR HDFLO.

The GENR HDFLO technique enhances feature selection by identifying clusters of correlated features rather than individual ones, improving predictive performance and understanding of data relationships. Each modality’s features are treated as separate clusters, applying GENR within them to account for interdependencies. The method integrates $L_{1}$ and $L_{2}$ regularizations (lasso and ridge) into an ENR framework, balancing sparsity and coefficient magnitude. The HDFLO algorithm combines Fick’s Law Optimization (FLO) and Dandelion Optimization (DO) to balance exploration and exploitation during optimization. DO simulates dandelion seed dispersion stages, while FLO’s Steady-State Operator (SSO) ensures efficient navigation through optimization stages.

Instead of simply selecting individual features, it identifies and selects groups or clusters of features that exhibit similar behavior or have high correlations. This leads to improved predictive performance and a better understanding of the underlying relationships in the data. Apply GENR to each modality separately by treating the features from each modality as a separate cluster and performing ENR within that cluster. This step helps to identify the most relevant features within each modality while considering their interdependencies.

L1 and L2 Regularization: $L_{2}$ regularization encompasses adding a regularization parameter $β$ multiplied by the sum of the squares of the weights to the loss function $L F$ . This method is known as Tikhonov Regularization. On the other hand, $L_{1}$ Regularization replaces the squared weights with the absolute value of weights. The mathematical representation formulated in Eq. (3),

LF=1nn∑j=1(v−ˆv)2L1=LF+βn∑j=1|uj|=1nn∑j=1(v−ˆv)2+βn∑j=1|uj|L2=LF+βn∑j=1u2j=1nn∑j=1(v−ˆv)2+βn∑j=1u2j<math display="block" altimg="eq-00023.gif"><mtable displaystyle="true" equalrows="false" equalcolumns="false" class="array"><mtr><mtd columnalign="right"><mi>L</mi><mi>F</mi></mtd><mtd columnalign="center"><mo>=</mo></mtd><mtd columnalign="left"><mfrac><mrow><mn>1</mn></mrow><mrow><mi>n</mi></mrow></mfrac><munderover accentunder="true" accent="true"><mrow><mo>∑</mo></mrow><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>n</mi></mrow></munderover><msup><mrow><mo stretchy="false">(</mo><mi>v</mi><mo>−</mo><mover accent="true"><mrow><mi>v</mi></mrow><mo>̂</mo></mover><mo stretchy="false">)</mo></mrow><mrow><mn>2</mn></mrow></msup></mtd></mtr><mtr><mtd columnalign="right"><msub><mrow><mi>L</mi></mrow><mrow><mn>1</mn></mrow></msub></mtd><mtd columnalign="center"><mo>=</mo></mtd><mtd columnalign="left"><mi>L</mi><mi>F</mi><mo>+</mo><mi>β</mi><munderover accentunder="true" accent="true"><mrow><mo>∑</mo></mrow><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>n</mi></mrow></munderover><mi>|</mi><msub><mrow><mi>u</mi></mrow><mrow><mi>j</mi></mrow></msub><mi>|</mi><mo>=</mo><mfrac><mrow><mn>1</mn></mrow><mrow><mi>n</mi></mrow></mfrac><munderover accentunder="true" accent="true"><mrow><mo>∑</mo></mrow><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>n</mi></mrow></munderover><msup><mrow><mo stretchy="false">(</mo><mi>v</mi><mo>−</mo><mover accent="true"><mrow><mi>v</mi></mrow><mo>̂</mo></mover><mo stretchy="false">)</mo></mrow><mrow><mn>2</mn></mrow></msup><mo>+</mo><mi>β</mi><munderover accentunder="true" accent="true"><mrow><mo>∑</mo></mrow><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>n</mi></mrow></munderover><mi>|</mi><msub><mrow><mi>u</mi></mrow><mrow><mi>j</mi></mrow></msub><mi>|</mi></mtd></mtr><mtr><mtd columnalign="right"><msub><mrow><mi>L</mi></mrow><mrow><mn>2</mn></mrow></msub></mtd><mtd columnalign="center"><mo>=</mo></mtd><mtd columnalign="left"><mi>L</mi><mi>F</mi><mo>+</mo><mi>β</mi><munderover accentunder="true" accent="true"><mrow><mo>∑</mo></mrow><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>n</mi></mrow></munderover><msubsup><mrow><mi>u</mi></mrow><mrow><mi>j</mi></mrow><mrow><mn>2</mn></mrow></msubsup><mo>=</mo><mfrac><mrow><mn>1</mn></mrow><mrow><mi>n</mi></mrow></mfrac><munderover accentunder="true" accent="true"><mrow><mo>∑</mo></mrow><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>n</mi></mrow></munderover><msup><mrow><mo stretchy="false">(</mo><mi>v</mi><mo>−</mo><mover accent="true"><mrow><mi>v</mi></mrow><mo>̂</mo></mover><mo stretchy="false">)</mo></mrow><mrow><mn>2</mn></mrow></msup><mo>+</mo><mi>β</mi><munderover accentunder="true" accent="true"><mrow><mo>∑</mo></mrow><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>n</mi></mrow></munderover><msubsup><mrow><mi>u</mi></mrow><mrow><mi>j</mi></mrow><mrow><mn>2</mn></mrow></msubsup></mtd></mtr></mtable></math>(3)

where,

$β$ represents the strength of regularization, the total number of rows are denoted as n, u represents weights, and j denotes iterations. v represents the ground truth values used during the training phase of the regression model, while

$\hat{v}$ represents the predicted values generated by the model. The variables

$n, m$ denote the total number of rows and columns, respectively, in the training dataset.

$L_{2}$ regularization is used in ridge regression whereas

$L_{1}$ regularization is used in lasso regression. As it incorporates both

$L_{2}$ and

$L_{1}$ regularization into the same expression, ENR serves as a bridge between lasso and ridge. The ENR measurements are given as follows,

Ridge Regression: Ridge regression incorporates $L_{2}$ regularization along with the Mean Squared Error (MSE) loss function. Its cost function $G (δ)$ is depicted in Eq. (4),

G(δ)=1nn∑j=1(v−ˆv)2+βm∑i=1u2i<math display="block" altimg="eq-00033.gif"><mi>G</mi><mo stretchy="false">(</mo><mi>δ</mi><mo stretchy="false">)</mo><mo>=</mo><mfrac><mrow><mn>1</mn></mrow><mrow><mi>n</mi></mrow></mfrac><munderover accentunder="true" accent="true"><mrow><mo>∑</mo></mrow><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>n</mi></mrow></munderover><msup><mrow><mo stretchy="false">(</mo><mi>v</mi><mo>−</mo><mover accent="true"><mrow><mi>v</mi></mrow><mo>̂</mo></mover><mo stretchy="false">)</mo></mrow><mrow><mn>2</mn></mrow></msup><mo>+</mo><mi>β</mi><munderover accentunder="true" accent="true"><mrow><mo>∑</mo></mrow><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>m</mi></mrow></munderover><msubsup><mrow><mi>u</mi></mrow><mrow><mi>i</mi></mrow><mrow><mn>2</mn></mrow></msubsup></math>(4)

where,

$\frac{1}{n} \sum_{j = 1}^{n} {(v - \hat{v})}^{2}$ denotes the loss function’s Mean Square Error (MSE) and

$β \sum_{i = 1}^{n} u_{i}^{2}$ denotes the penalty (

$L_{2}$ regularization). Gradient descent is used by all regression algorithms to determine the optimal or minimum weights and biases. This is accomplished by first calculating the partial derivative of this cost function about bias and weights.

Lasso Regression: Lasso regression utilizes $L_{1}$ regularization, follows similar processes as ridge regression, and employs the same MSE loss function. The cost function $G (δ)$ in lasso regression is represented by Eq. (5).

G(δ)=1nn∑j=1(v−ˆv)2+βm∑i=1|ui|<math display="block" altimg="eq-00039.gif"><mi>G</mi><mo stretchy="false">(</mo><mi>δ</mi><mo stretchy="false">)</mo><mo>=</mo><mfrac><mrow><mn>1</mn></mrow><mrow><mi>n</mi></mrow></mfrac><munderover accentunder="true" accent="true"><mrow><mo>∑</mo></mrow><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>n</mi></mrow></munderover><msup><mrow><mo stretchy="false">(</mo><mi>v</mi><mo>−</mo><mover accent="true"><mrow><mi>v</mi></mrow><mo>̂</mo></mover><mo stretchy="false">)</mo></mrow><mrow><mn>2</mn></mrow></msup><mo>+</mo><mi>β</mi><munderover accentunder="true" accent="true"><mrow><mo>∑</mo></mrow><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>m</mi></mrow></munderover><mi>|</mi><msub><mrow><mi>u</mi></mrow><mrow><mi>i</mi></mrow></msub><mi>|</mi></math>(5)

where,

$β \sum_{i = 1}^{m} | u_{i} |$ is the

$L_{1}$ regularization,

$\frac{1}{n} \sum_{j = 1}^{n} {(v - \hat{v})}^{2}$ denotes the loss function’s MSE

ENR: In this network, the cost function combines the MSE loss function with both $L_{2}$ and $L_{1}$ regularization, expressed in Eq. (6),

G(δ)=1nn∑j=1(v−ˆv)2+sβm∑i=1|ui|+1−s2βm∑i=1|u2i|<math display="block" altimg="eq-00045.gif"><mi>G</mi><mo stretchy="false">(</mo><mi>δ</mi><mo stretchy="false">)</mo><mo>=</mo><mfrac><mrow><mn>1</mn></mrow><mrow><mi>n</mi></mrow></mfrac><munderover accentunder="true" accent="true"><mrow><mo>∑</mo></mrow><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>n</mi></mrow></munderover><msup><mrow><mo stretchy="false">(</mo><mi>v</mi><mo>−</mo><mover accent="true"><mrow><mi>v</mi></mrow><mo>̂</mo></mover><mo stretchy="false">)</mo></mrow><mrow><mn>2</mn></mrow></msup><mo>+</mo><mspace width=".17em" class="thinspace"></mspace><mi>s</mi><mi>β</mi><munderover accentunder="true" accent="true"><mrow><mo>∑</mo></mrow><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>m</mi></mrow></munderover><mi>|</mi><msub><mrow><mi>u</mi></mrow><mrow><mi>i</mi></mrow></msub><mi>|</mi><mo>+</mo><mfrac><mrow><mn>1</mn><mo>−</mo><mi>s</mi></mrow><mrow><mn>2</mn></mrow></mfrac><mi>β</mi><munderover accentunder="true" accent="true"><mrow><mo>∑</mo></mrow><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>m</mi></mrow></munderover><mi>|</mi><msubsup><mrow><mi>u</mi></mrow><mrow><mi>i</mi></mrow><mrow><mn>2</mn></mrow></msubsup><mi>|</mi></math>(6)

where,

$s β \sum_{i = 1}^{m} | u_{i} | L_{1}$ regularization,

$\frac{1 - s}{2} β \sum_{i = 1}^{m} | u_{i}^{2} | L_{2}$ regularization, and

$\frac{1}{n} \times \sum_{j = 1}^{n} {(v - \hat{v})}^{2}$ represents the MSE loss function. The hyperparameter

$L_{1}$ ratio is specified as s, controls the behavior of elastic net regression, which combines aspects of both lasso and ridge regression. If

$s = 0.5$ , the elastic net acts similarly to the Ridge and Lasso. Elastic net approximates the ridge regression when s is reduced to zero and resembles ridge regression when s is zero, it completely. Elastic net behaves similarly to Lasso when s is set to 1, but when s is set to 1, it completely resembles Lasso regression.²⁵ Consider a multimodal dataset with three modalities: audio (A), video (V), and text (T). Each modality has its own set of features, denoted by

$A = [a_{1}, a_{2}, \dots, a_{i}]$ ,

$V = [v_{1}, v_{2}, \dots, v_{i}]$ , and

$T = [t_{1}, t_{2}, \dots, t_{i}]$ , respectively. It formulates the GENR (G) as Eq. (7),

where, the loss function represents the specific regression loss based on feature selection (e.g., mean squared error).

$β a_{i}$ ,

$β v_{i}$ and

$β t_{i}$ are the regression coefficients associated with the features from audio, video, and text modalities respectively.

$λ_{1}$ is the regularization parameter that controls the

$L_{1}$ (Lasso) penalty term and encourages sparsity by pushing the coefficients to zero.

$λ_{2}$ is the regularization parameter that controls the

$L_{2}$ (Ridge) penalty term and helps control the overall magnitude of the coefficients. The loss function is minimized by the HDFLO model.

HDFLO: The algorithm integrates FLO and DO, combining their update rules to optimize exploration and exploitation. DO mimics dandelion seed flight stages such as rising in spirals or drifting locally, adjusting globally, and landing randomly to grow. Meanwhile, FLO includes an SSO ensuring a balance between exploration and exploitation. By incorporating FLO’s SSO into DO, the algorithm achieves enhanced capability in navigating diverse optimization stages effectively, avoiding local optima stagnation, and ensuring robust exploration-exploitation trade-offs. A detailed explanation of the mathematical model behind this new meta-heuristic algorithm, which draws inspiration from the optimal reproduction location of dandelion seeds as they mature, is provided. In the initialization phase DO randomly generates a candidate solution which is given in Eq. (8),

X i, j = r a n d \times (u b j - l b j) + l b j i = 1, 2, \dots, p o p; j = 1, 2, \dots, D i m <math display="block" altimg="eq-00062.gif"><msub><mrow><mi>X</mi></mrow><mrow><mi>i</mi><mo>,</mo><mi>j</mi></mrow></msub><mo>=</mo><mi>r</mi><mi>a</mi><mi>n</mi><mi>d</mi><mo>\times</mo><mo stretchy="false">(</mo><mi>u</mi><msub><mrow><mi>b</mi></mrow><mrow><mi>j</mi></mrow></msub><mo>-</mo><mi>l</mi><msub><mrow><mi>b</mi></mrow><mrow><mi>j</mi></mrow></msub><mo stretchy="false">)</mo><mo>+</mo><mi>l</mi><msub><mrow><mi>b</mi></mrow><mrow><mi>j</mi></mrow></msub><mspace width="1em" class="quad"></mspace><mi>i</mi><mo>=</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mo>\dots</mo><mo>,</mo><mi>p</mi><mi>o</mi><mi>p</mi><mo>;</mo><mspace width="1em" class="quad"></mspace><mi>j</mi><mo>=</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mo>\dots</mo><mo>,</mo><mi>D</mi><mi>i</mi><mi>m</mi></math> (8)

where,

$X_{i, j}$ represents the candidate solution,

$r a n d$ is the random number, size of the population and dimension of the variable are represented as

$p o p$ and

$D i m$ .

$u b$ and

$l b$ are the upper bound and lower bound of the seed position.

The fitness value $(F (X_{i}))$ creates the fitness function of the ith seed in the population which is taken to minimize the loss function. The fitness function ( $F i t n e s s_{f u n c t i o n}$ ) is expressed in Eq. (9),

Fitnessfunction=Min(1nm∑j=1(v−ˆv)2)<math display="block" altimg="eq-00071.gif"><mi>F</mi><mi>i</mi><mi>t</mi><mi>n</mi><mi>e</mi><mi>s</mi><msub><mrow><mi>s</mi></mrow><mrow><mi>f</mi><mi>u</mi><mi>n</mi><mi>c</mi><mi>t</mi><mi>i</mi><mi>o</mi><mi>n</mi></mrow></msub><mo>=</mo><mi>M</mi><mi>i</mi><mi>n</mi><mfenced separators="" open="(" close=")"><mrow><mfrac><mrow><mn>1</mn></mrow><mrow><mi>n</mi></mrow></mfrac><munderover accentunder="true" accent="true"><mrow><mo>∑</mo></mrow><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>m</mi></mrow></munderover><msup><mrow><mo stretchy="false">(</mo><mi>v</mi><mo>−</mo><mover accent="true"><mrow><mi>v</mi></mrow><mo>̂</mo></mover><mo stretchy="false">)</mo></mrow><mrow><mn>2</mn></mrow></msup></mrow></mfenced></math>(9)

where, v represents the ground truth,

$\hat{v}$ represents the predicted value by the model,

$n, m$ denotes the number of rows and columns in the training dataset.

$M i n$ represents the minimization of the loss function.

Rising stage: In this stage, the weather conditions along with the speed of wind jointly find the dandelion seed’s height. In the search space, dandelion seeds are blown to various locations, rising higher and scattering farther with stronger wind, following a spiral motion influenced by wind speed and vortex adjustments, which is expressed in Eq. (10),

X t + 1 i = X t i + α * s x * s y * ln (Y) * (X t s - X t i) <math display="block" altimg="eq-00075.gif"><msubsup><mrow><mi>X</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msubsup><mo>=</mo><msubsup><mrow><mi>X</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi>t</mi></mrow></msubsup><mo>+</mo><mi>α</mi><mo>*</mo><msub><mrow><mi>s</mi></mrow><mrow><mi>x</mi></mrow></msub><mo>*</mo><msub><mrow><mi>s</mi></mrow><mrow><mi>y</mi></mrow></msub><mo>*</mo><mo>ln</mo><mo stretchy="false">(</mo><mi>Y</mi><mo stretchy="false">)</mo><mo>*</mo><mo stretchy="false">(</mo><msubsup><mrow><mi>X</mi></mrow><mrow><mi>s</mi></mrow><mrow><mi>t</mi></mrow></msubsup><mo>-</mo><msubsup><mrow><mi>X</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi>t</mi></mrow></msubsup><mo stretchy="false">)</mo></math> (10)

where, the seed position at iteration t is expressed as

$X_{i}^{t}$ , the randomly selected position at t is denoted as

$X_{s}^{t}$ ,

$ln (Y)$ denotes a lognormal distribution obey in

$μ = 0$ ,

$σ^{2} = 1$ , the adaptive parameter is denoted as

$α$ , seed’s lift component coefficients are represented as

$s_{x}$ and

$s_{y}$ . The diffusion coefficient represents the variability or exploration level of the algorithm, and the concentration gradient is related to the fitness landscape or objective function. Table 2 gives the pseudocode for the HDFLO algorithm.

**Table 2. Pseudocode for HDFLO algorithm.**
Input: pop, $T, D i m$ and C.
Output: $X_{b e s t}$ (best position of seed)
• Initial position of dandelion seeds.
• Compute fitness function as, $F i t n e s s_{f u n c t i o n} = M i n (\frac{1}{n} \sum_{j = 1}^{m} {(v - \hat{v})}^{2})$
if $(t \leq T)$ do
if $i = 1$ to pop do
if $r a n d n < 1.5$
Compute the TF by eqn. (13)
If $T F < 0.9$
Update the position by eqn. (15)
Else if
Update dandelion seeds
else Rising stage rainy day
Update dandelion seeds by multiplying flus with a position as,
$X_{i}^{t + 1} = X_{i}^{t} + a * s_{x} * s_{y} * ln Y * (X_{s}^{t} - X_{i}^{t})$
end for
if $i = 1$ to pop do
update descending stage position
end for
if $i = 1$ to pop do
Update landing stage position
end for
if $i = 1$ to pop do
arrange seeds based on fitness values
end for
if $f (X_{e l i t e}) < f (X_{b e s t})$ then
$X_{b e s t} = X_{e l i t e}$
$f_{b e s t} = f (X_{e l i t e})$ // Best position and fitness value

Descending stage: In this phase, the dandelion population travels to the appropriate location for reproduction which is reflected by the mean location in the rising stage. The descending phase ( $X_{i}^{t + 1}$ ) is mathematically expressed in Eq. (11),

X t + 1 i = X t i - a * β t * (X m e a n_t - α * β t * X t i) <math display="block" altimg="eq-00106.gif"><msubsup><mrow><mi>X</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msubsup><mo>=</mo><msubsup><mrow><mi>X</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi>t</mi></mrow></msubsup><mo>-</mo><mi>a</mi><mo>*</mo><msub><mrow><mi>β</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>*</mo><mo stretchy="false">(</mo><msub><mrow><mi>X</mi></mrow><mrow><mi>m</mi><mi>e</mi><mi>a</mi><mi>n</mi><mtext>_</mtext><mi>t</mi></mrow></msub><mo>-</mo><mi>α</mi><mo>*</mo><msub><mrow><mi>β</mi></mrow><mrow><mi>t</mi></mrow></msub><mo>*</mo><msubsup><mrow><mi>X</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi>t</mi></mrow></msubsup><mo stretchy="false">)</mo></math> (11)

where, the seed position is expressed as

$X_{i}^{t}$ , an adaptive parameter is denoted as

$α$ , Brownian motion is expressed as

$β_{t}$ and the mean location of the population in ith repetition is denoted as

$X_{m e a n_t}$ .

Landing stage: In this local neighborhood development is achieved to get the global optimum where local exploitation is taken by the current elite information. The landing stage population ( $X_{i}^{t + 1}$ ) is expressed in Eq. (12),

Xt+1i=Xelite+λ∗a∗(Xelite−Xti∗δ):δ=2tT<math display="block" altimg="eq-00112.gif"><msubsup><mrow><mi>X</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msubsup><mo>=</mo><msub><mrow><mi>X</mi></mrow><mrow><mi>e</mi><mi>l</mi><mi>i</mi><mi>t</mi><mi>e</mi></mrow></msub><mo>+</mo><mi>λ</mi><mo>∗</mo><mi>a</mi><mo>∗</mo><mo stretchy="false">(</mo><msub><mrow><mi>X</mi></mrow><mrow><mi>e</mi><mi>l</mi><mi>i</mi><mi>t</mi><mi>e</mi></mrow></msub><mo>−</mo><msubsup><mrow><mi>X</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi>t</mi></mrow></msubsup><mo>∗</mo><mi>δ</mi><mo stretchy="false">)</mo><mo>:</mo><mi>δ</mi><mo>=</mo><mfrac><mrow><mn>2</mn><mi>t</mi></mrow><mrow><mi>T</mi></mrow></mfrac></math>(12)

where, each iteration is represented as t, the seed position at iteration t is expressed as

$X_{i}^{t}$ ,

$δ$ indicates the linear coefficient. the seed best location is denoted as

$X_{e l i t e}$ , the maximum number of iterations is denoted as T,

$λ$ denotes a function between [0,2].

SSO of FLO: Successful optimization algorithms hinge on transitioning between exploration and exploitation effectively. Therefore, the Transfer Function (TF) is computed to smoothly navigate between the exploration and exploitation phases, improving adaptability and performance, as provided in Eq. (13).

TF=sinh(tT)c1<math display="block" altimg="eq-00117.gif"><mi>T</mi><mi>F</mi><mo>=</mo><mo>sinh</mo><msup><mrow><mfenced separators="" open="(" close=")"><mrow><mfrac><mrow><mi>t</mi></mrow><mrow><mi>T</mi></mrow></mfrac></mrow></mfenced></mrow><mrow><msub><mrow><mi>c</mi></mrow><mrow><mn>1</mn></mrow></msub></mrow></msup></math>(13)

where,

$c_{1} = 0.5$ . The final stage in the optimization search is the exploitation phase, where molecules update their positions

$(X_{i}^{t + 1})$ for stability, as presented in Eq. (14),

X t + 1 i = X t S + Q t g \times X t i + Q t g \times (M S t i \times X t S - X t i) <math display="block" altimg="eq-00120.gif"><msubsup><mrow><mi>X</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msubsup><mo>=</mo><msubsup><mrow><mi>X</mi></mrow><mrow><mi>S</mi></mrow><mrow><mi>t</mi></mrow></msubsup><mo>+</mo><msubsup><mrow><mi>Q</mi></mrow><mrow><mi>g</mi></mrow><mrow><mi>t</mi></mrow></msubsup><mo>\times</mo><msubsup><mrow><mi>X</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi>t</mi></mrow></msubsup><mo>+</mo><msubsup><mrow><mi>Q</mi></mrow><mrow><mi>g</mi></mrow><mrow><mi>t</mi></mrow></msubsup><mo>\times</mo><mo stretchy="false">(</mo><mi>M</mi><msubsup><mrow><mi>S</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi>t</mi></mrow></msubsup><mo>\times</mo><msubsup><mrow><mi>X</mi></mrow><mrow><mi>S</mi></mrow><mrow><mi>t</mi></mrow></msubsup><mo>-</mo><msubsup><mrow><mi>X</mi></mrow><mrow><mi>i</mi></mrow><mrow><mi>t</mi></mrow></msubsup><mo stretchy="false">)</mo></math> (14)

where

$X_{i}^{t}$ denotes the position of the paricle,

$X_{S}^{t}$ denotes the steady state location,

$M S_{i}^{t}$ refers to the motion step and

$Q_{g}^{t}$ represents the relative quantity of the region.

This optimization algorithm, inspired by dandelion seed dispersal, initializes a population with random seed positions. Each seed’s fitness is evaluated based on an objective function, iteratively refining positions over generations. The algorithm balances exploration and exploitation inspired by FO using a transfer function, adjusting seed exploration and exploitation strategies. During the rising stage, seeds adjust positions influenced by parameters like wind speed and direction, simulating natural dispersal patterns. In the descending stage, seeds move towards optimal locations identified in previous stages, and in the landing stage, positions are further refined for optimal growth conditions. The algorithm continuously updates the best seed position based on fitness, converging towards an optimal solution by the end of iterations, effectively navigating diverse optimization stages while avoiding local optima.

Thus, the selected features are based on multimodal data where the loss function of GENR is minimized with HDFLO to efficiently extract relevant features. Then these features are taken to classify the sentiments as three dimensions.

3.4. Emotional states classification using MWCNN

The selected features are classified as three types of emotional states such as valence, arousal, and trustworthiness from audio, video, and text. For classification, a formulation is achieved to connect convolution and pooling with multimodal data analysis. The MWCNN classifier integrates CNN and spectral analyses, capturing spatial and frequency domain features. By using wavelet transforms, this model extends traditional CNN architecture, enhancing the feature extraction and classification process.

3.4.1. Classification based on emotional states

A CNN is a variant of a Neural Network (NN) to sparsely connect deep networks. In conventional NN each input in one layer is connected with the next layer but in the proposed classifier the use of an activation function and fully connected layer, CNN introduces a convolution/pooling layer that connects only the local respective field around each input.

A fusion of CNNs and spectral analyses is employed to address texture classification challenges. CNNs process textures directly to capture spatial statistics, while spectral analysis transforms textures into frequency domains for scale-invariant features. This unified approach integrates spatial and spectral information within a single model. By extending the traditional CNN architecture with multiresolution analysis through wavelet transforms, the pooling and convolution layers mimic aspects of spectral analysis. This innovative model, termed wavelet CNNs, combines CNN strengths with spectral analysis for comprehensive texture feature extraction. Figure 3 illustrates the configuration of the MWCNN-based classifier.

Fig. 3. The architecture of MWCNN based classifier.

Convolutional layer: For the n component input vector, a convolutional layer produces the output vector of the same number of components. The output vector ( $Y_{i}$ ) is expressed as Eq. (15),

Y i = \sum j \in N i w j x j <math display="block" altimg="eq-00126.gif"><msub><mrow><mi>Y</mi></mrow><mrow><mi>i</mi></mrow></msub><mo>=</mo><munder><mrow><mo>\sum</mo></mrow><mrow><mi>j</mi><mo>\in</mo><msub><mrow><mi>N</mi></mrow><mrow><mi>i</mi></mrow></msub></mrow></munder><msub><mrow><mi>w</mi></mrow><mrow><mi>j</mi></mrow></msub><msub><mrow><mi>x</mi></mrow><mrow><mi>j</mi></mrow></msub></math> (15)

where,

$x_{j} = x_{0}, x_{1}, \dots, x_{n - 1}$ is the n number of the input vector,

$w_{j}$ is a weight,

$N_{i}$ is the set of indices in the local respective field at input vector and weight. The wight includes the bias by having a constant input of 1. Thus, the output is the addition of weighted sum and constant.

CNNs achieve translation invariance in the image space and reduce the parameter count by parameter sharing. The definition of $Y_{i}$ is essentially the result of convolving $x_{j}$ with a filtering kernel $w_{j}$ hence it is referred to as a convolution layer. Consequently, the output (y) is expressed as Eq. (16),

y = x * w <math display="block" altimg="eq-00133.gif"><mi>y</mi><mo>=</mo><mi>x</mi><mo>*</mo><mi>w</mi></math> (16)

where, w is the set of weights for the same input, and the output results as a concatenated vector.

Pooling layer: Pooling layers are commonly employed right after convolution layers to simplify data representation. Although max pooling finds extensive use in CNN applications, average pooling is better suited for feature extraction. Consequently, it focusses lies on average pooling, which offers the advantage of revealing the association with multiresolution analysis. The output of pooling is expressed as Eq. (17),

y = (x * P) ↓ p <math display="block" altimg="eq-00134.gif"><mi>y</mi><mo>=</mo><mo stretchy="false">(</mo><mi>x</mi><mo>*</mo><mi>P</mi><mo stretchy="false">)</mo><mi>↓</mi><mi>p</mi></math> (17)

where,

$P = \frac{1}{p}, \dots, \frac{n}{p}$ is the support of pooling which represents the average filter to reduce the number of outputs by taking a pairwise average of half of the input, the input vector is represented as x and y is the output vector. Average pooling involves convolution through P followed by down sampling p.

3.4.2. Generalized convolution and pooling

By combining the output of convolution and average pooling the generalized form is taken as Eq. (18),

y = (x * k) ↓ p <math display="block" altimg="eq-00136.gif"><mi>y</mi><mo>=</mo><mo stretchy="false">(</mo><mi>x</mi><mo>*</mo><mi>k</mi><mo stretchy="false">)</mo><mi>↓</mi><mi>p</mi></math> (18)

where, x is the input vector and y is the output vector and k is the wavelet function where

$k = w$ with

$p = 1$ for convolution and

$k = p$ with

$p > 1$ for pooling and

$k = w * P$ with

$p > 1$ for generalized form. If it has the pair of high pass and low pass filter, the data part is decomposed as low and high frequency with

$p = 2$ . Traditional CNNs are known for utilizing low-frequency components and disregarding the wavelet.

Implementation: The network structure is modeled after VGG-19, selected for its effectiveness in texture feature extraction, using $3 \times 3$ convolutional kernels with $1 \times 1$ padding to preserve input size. Convolution layers with increased stride are utilized; incorporating $1 \times 1$ padding and a stride of two results in output size reduction, replacing max-pooling without compromising accuracy. To align with the size reduction in multiresolution analysis, decomposed images are integrated with feature maps. An energy layer preceding fully connected layers enhances performance with fewer parameters. The wavelet CNN model comprises nine convolution layers, aligned with decomposition levels, featuring an energy layer followed by three fully connected layers. Input size constraints ( $32 \times 32$ ) necessitate training images to be scaled to $256 \times 256$ , then randomly cropped and flipped for diversity, effectively mitigating overfitting.

In contrast, proposed wavelet CNNs incorporate all components, including the high-frequency components so that no information from the input x is lost, aligning with the principles of multimodal analysis. The classified output is then compared with the existing methods to find its performance.

This research work gathered input from the MuSe-CaR dataset to tackle data imbalance in sentiment analysis across modalities by employing a comprehensive approach. This involves combining DSMOTE with an ECGAN for emotion recognition using audio, text, and visual data. The balanced data is then processed through GENR with an HDFLO to select relevant features for sentiment analysis. Subsequently, a MWCNN is employed to accurately classify emotion states. By integrating these techniques, the research aims to overcome challenges associated with diverse data types and provide a robust framework for MSA in the context of car reviews.

4. Experiment Results

MuSe-CaR dataset maintains the content and context largely without breaking into equally sized segments which is better than other large datasets. This section also takes an analysis of python implementation in terms of three dimensions such as arousal, valence, and trustworthiness against some conventional approaches.^{15,16,17,18,19,20,21,22,23,24} The performance metrics such as accuracy, recall, and precision are used to compare the proposed feature selection method of GENR HDFLO in contrast with existing methods.^26,27,28,29

4.1. Experimental setup

The experiments utilized Anaconda Navigator Spyder with python 3.10 on an Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz, 16.0GB RAM, and Windows 10. Following network design, factors impacting performance were analyzed, including epochs, iterations, and learning rate. Set at 0.001 and 100 respectively, experiments ran for 1000 epochs with 10 timesteps. Using a patch size of $32 \times 32$ enabled efficient comparison with existing methods. The proposed GENR model, implemented with the TensorFlow library, underwent evaluation based on different features in a multimodal approach. The MuSe-CaR dataset was split into a 60:20 ratio for training, development, and test sets to ensure robust evaluation.

Data preprocessing: Gather data from the MuSe-CaR dataset, which includes multimodal data such as audio, video, and text reviews. For initial pre-processing, handle missing data by removing incomplete entries. Since the annotation for MuSe-CaR is dense (4 frames per second), split each video into segments with a window size of 200 frames (50s) and a hop size of 100 frames (25s). This segmentation enriches training samples and facilitates model convergence. A moving average filter is utilized to smooth the input signal of each sliding filter frame for better intelligibility.

Model Training: The developed scheme is executed using python 3.10. The network structure is based on VGG-19 for effective texture feature extraction, using $3 \times 3$ convolutional kernels with $1 \times 1$ padding. The wavelet CNN model includes nine convolution layers, an energy layer, and three fully connected layers. Training images are scaled to $256 \times 256$ , then randomly cropped and flipped for diversity.

Key hyperparameters: Learning rate 0.001, 1000 epochs, and Adam optimizer. Batch normalization is used before activation layers, with ReLU as the activation function.

4.2. Dataset description

MuSe-CaR is a comprehensive dataset of multimodal data collected in real-world settings, aimed at understanding emotional engagement in product reviews, particularly automotive ones. It ensures high-quality voice and video recordings, offering valuable social media content. The sample video image of the MuSe-CaR dataset is shown in Fig. 4.

Fig. 4. Sample images from the Videos of the MuSe-CaR dataset.

4.3. Density estimation of the dimensions

Figure 5 presents a comparison of the distributions for the manually labeled annotations of arousal, valence, and trustworthiness following feature selection.

Fig. 5. Density estimation of the dimensions.

While arousal demonstrates a near-perfect Gaussian distribution, valence exhibits a positive skewness, leaning towards the positive end of the spectrum. Trustworthiness, on the other hand, displays a highly peaked distribution with a strong left-skew. The model aims to analyze the phenomenon of feature selection due to the high-density distribution observed across all three dimensions.

4.4. The feature set of each modality in the MuSe-CaR dataset

Table 3 illustrates the feature set of each modality with the corresponding arousal and valence dimension where “A” is the representation of audio, “V” is the representation of video, and “T” is the representation of text.

**Table 3. Feature set of each modality in MuSe-CaR dataset.**
Features	Modality	Arousal	Valence
Energy	A	0.4684	0.2633
Pitch	A	0.4596	0.2484
MFCC	A	0.4313	0.2646
Wav2vec	A	0.4818	0.333
VGGFace	V	0.4023	0.2241
HOG	V	0.4653	0.1059
SeNetFace	V	0.4678	0.1543
ResNetFace	V	0.4311	0.1344
BERT	T	0.4325	0.5624
RoBERT	T	0.4256	0.6132
ALBERT	T	0.4356	0.5532
BOW	T	0.4567	0.5689
Word2vec	T	0.4675	0.6732

The analysis reveals several key findings: Firstly, in terms of arousal, audio features demonstrate higher effectiveness compared to visual and text features. Secondly, for valence, the text modality exhibits significantly superior performance compared to the other two modalities. Lastly, the visual modality generally yields the least favorable results. The researchers speculate that the dominance of audio features in arousal assessment stems from the reliance on speech-related aspects such as intonation and tone, whereas valence assessment is primarily influenced by verbal content.

4.5. Confusion matrix of three-dimension prediction using classifier

For this experiment, it focused on features from acoustic, linguistic, and visual modalities. Figure 6 displays the confusion matrix derived from using features to forecast emotion categories in arousal, valence, and trustworthiness.

Fig. 6. Confusion matrix of prediction using classifier. (a) Valence. (b) Arousal. and (c) Trustworthiness.

When comparing the confusion matrix obtained from a three-dimensional representation with the corresponding fixed length representation, there is an improvement in the weighting along the diagonal of the confusion matrix. The findings indicate that, overall, valence yields better results than arousal for this task, and trustworthiness performs better than arousal as well. Consequently, predicting arousal proves to be more challenging in detecting audio modality compared to other modalities.

4.6. Continuous prediction analysis with frequency distribution

The MuSe-CaR dataset predicts sentiment dimensions (valence, arousal, and trustworthiness) in a continuous manner with time. The sentiment analysis has the emotional component of valence which is often used interchangeably.

The process of manually annotating continuous emotions by humans often results in disagreements due to variations in perception and reaction time. To mitigate these issues, a feature selection model is employed. Various selection methods are discussed in existing literature, and this advancement has also sparked new challenges. In Fig. 7, a feature selection method is depicted which utilizes the distribution of proposed features from a minimum of five different ratings. Additionally, the model takes into account the diverse reaction times observed during the process.

Fig. 7. Continuous prediction analysis with frequency distribution. (a) Valence. (b) Arousal. (c) Trustworthiness.

4.7. Comparison of unimodal and multimodal using 6-fold cross validation

Figure 8 denotes the results of a 6-fold cross-validation experiment designed to compare two emotion recognition models: unimodal and multimodal. The unimodal is trained solely on video frame data, while the multimodal model capitalizes on information from three different modalities (likely visual, audio, and another source). The experiment employed a 6-fold cross-validation technique, where the data is split into six sections. Each section is then used for testing, while the remaining sections are employed for training. This process is repeated six times, ensuring a robust evaluation of both models.

The key finding from Fig. 8 is the clear superiority of the multimodal model. It consistently outperforms the unimodal model across all three emotional dimensions: valence, arousal, and trustworthiness. This indicates that incorporating additional data modalities significantly enhances the model’s ability to accurately detect emotions compared to relying solely on video information. Interestingly, the results also suggest that trustworthiness might be a more readily detectable emotion, as it exhibits the highest accuracy across both models.

4.8. Comparison of the proposed MSA system against existing sentiment analysis systems

The sentiment analysis with the MuSe-CaR dataset provides three dimensions (Valance, arousal, and trustworthiness) to analyze the system. In this section, some of the sentiment analysis systems such as Multihead attention Network,¹⁵ attention enhanced Recurrent Model,¹⁶ MuSe-Toolbox.¹⁷

Sentiment Analysis of YouTube Comments,²⁰ LSTM-RNN,²¹ and temporal convolutional network²⁴ are taken to compare with proposed MSA systems.

Figure 9, shows that the proposed system attains high output to find the range of emotions as the dimensions valence, arousal, and Valence. The valence, arousal, and trustworthiness attain 0.6245, 0.723, and 0.695 high outcomes which is a 20% higher rate than other existing methods. Thus, the recent researches are motivated by multimodal data for sentiment analysis. The effectiveness of the proposed model is due to several key factors. Initially, the system uses DSMOTE combined with ECGAN to address data imbalance, which ensures the generation of high-quality, diverse synthetic samples. Subsequently, feature selection is optimized using GENR HDFLO, which captures relevant features and their interactions, thus improving predictive performance. Finally, the MWCNN classifier, incorporating wavelet transforms and robust convolutional architecture, accurately classifies emotional states from multimodal data. These advancements contribute to the system’s superior performance in valence, arousal, and trustworthiness, achieving up to 20% better results compared to other methods.

Fig. 9. Comparison of proposed MSA system against existing sentiment analysis systems. (a) Valence. (b) Arousal. (c) Trustworthiness.

4.9. Comparison against existing sentiment analysis systems

The classification performance is improved by limiting the large amount of data in the dataset. Thus, the feature selection approach is motivated to choose a set of relevant features. This research work uses the GENR HDFLO based selection method. GENR is applied to multimodal data, such as audio, video, and text by using recursive feature elimination to determine the appropriate number of features which is optimized to reduce loss function using HDFLO.

The existing approaches Principal Component Analysis (PCA),²⁶ Feature Weight-based Decision Tree algorithm (FWDT),²⁷ Deep Convolutional Network (DCN),²⁸ and Attention-based Bidirectional Convolutional Deep Model (ABCDM)²⁹ are taken to compare with the proposed GENR HDFLO based feature selection method in terms of accuracy, precision and recall. Figure 10, shows that the proposed GENR HDFLO method obtains 0.9965 accuracy, 0.9945 recall, and 0.9966 precision which are higher than the existing methods. The proposed GENR HDFLO model improves classification across a wide range of dimensions. Its superior performance compared to other methods is due to its comprehensive and innovative approach to handling multimodal data and addressing class imbalance. By using DSMOTE with ECGAN for data augmentation, GENR HDFLO for feature selection, and MWCNN for classification, the model effectively captures and analyzes precise emotional states from diverse inputs, resulting in more accurate and reliable sentiment analysis.

Fig. 10. Comparison of GENR HDFLO based feature selection against existing methods. (a) Accuracy. (b) Recall. (c) Precision.

Table 4 presents a performance comparison between analyses conducted without and with DSMOTE. Without DSMOTE, the model obtained an accuracy of 82.76%, recall of 80.27%, and precision of 80.46%. Introducing DSMOTE significantly improves the performance, resulting in a notable increase in accuracy to 94.76%, recall to 91.62%, and precision to 90.26%, respectively. This demonstrates a substantial percentage increase in model performance across all metrics with the incorporation of DSMOTE.

**Table 4. Performance comparisons without DSMOTE and with DSMOTE.**
Performance metrics	Analysis without DSMOTE	Analysis with DSMOTE
Accuracy	0.8276	0.9476
Recall	0.8027	0.9162
Precision	0.8046	0.9026

4.10. Comparison of feature selection approaches

In this section, shows the comparison of the proposed feature selection approach with existing models like RAAW,¹⁵ MAN,¹⁷ and Ensemble models,¹⁹ Table 5 compares different feature selection methods based on their performance metrics. RAAW achieved an accuracy of 82.76%, recall of 80.28%, and precision of 81.74%. MAN demonstrated improved performance with 84.75%, 82.87%, and 82.54% in accuracy, recall, and precision, respectively. Ensemble models yielded lower metrics at 74.62%, 72.94%, and 71.63%.

**Table 5. Comparison of feature selection methods.**
	Performance metrics
Methods	Accuracy	Recall	Precision
RAAW¹⁵	0.8276	0.8028	0.8174
MAN¹⁷	0.8475	0.8287	0.8254
Ensemble models¹⁹	0.7462	0.7294	0.7163
GENR HDFLO (proposed)	0.9476	0.9162	0.9026

The proposed method, GENR HDFLO, outperformed all, achieving a significantly higher accuracy of 94.76%, recall of 91.62%, and precision of 90.26%. GENR introduces a sophisticated feature selection mechanism, refining traditional ENR. This approach overcomes sparsity limitations, allowing for a more precise and adaptive model. By incorporating granular selection, GENR enhances the regularization process, leading to improved accuracy and robustness, particularly beneficial in sentiment analysis tasks with complex textual data.

Table 6 presents the performance analysis of the developed GENR HDFLO model based on different learning rates. As the learning rate decreases from 0.1 to 0.00001, there are changes in the model’s performance metrics. Table 6 demonstrates that the developed GENR HDFLO scheme achieved the optimal performance with a learning rate of 0.001, showing higher accuracy, recall, and precision compared to other learning rates. This finding suggests that a learning rate of 0.001 optimally balances the model’s ability to learn from the data, leading to improved overall performance in terms of classification accuracy and precision.

**Table 6. Performance analysis of the proposed model concerning learning rate.**
Model	Learning rate	Accuracy	Recall	Precision
GENR HDFLO (proposed)	0.1	0.8912	0.8573	0.8421
	0.01	0.9045	0.8673	0.8592
	0.001	0.9476	0.9162	0.9026
	0.0001	0.9228	0.8841	0.8721
	0.00001	0.9389	0.8943	0.8842

The developed model is evaluated across various patch sizes to identify the optimal setting, with $32 \times 32$ emerging as the most effective size. Figure 11 presents the comparative results achieved with different patch sizes, highlighting the performance variations observed across the range of sizes tested. This analysis underscores the significance of patch size selection in influencing model outcomes and showcases the superiority of the $32 \times 32$ patch size based on the experimental findings.

Fig. 11. Classification results for various patch sizes.

The accuracy graph depicted in Fig. 12 provides valuable insight into the learning progress of a model over training epochs. This visualization illustrates the model’s accuracy improvement with each iteration during training. Ideally, the graph shows a steady increase in accuracy as the model learns from the training data.

The proposed GENR HDFLO model demonstrates superior performance compared to existing models, as shown by the metrics in Table 7. It achieves an exceptionally low Mean Absolute Error (MAE) of 0.451, which is significantly lower than the MAE of HyCon (0.71) and AdaMoW (0.69), indicating high precision in predictions. The model also boasts an impressive F1-score of 94.21%, surpassing all other models and reflecting an excellent balance between precision and recall. This high F1-score highlights the model’s effectiveness in accurately identifying positive instances. Additionally, the model’s accuracy of 94.76% is notably high, demonstrating robust performance. This combination of a low error rate, high F1-score, and substantial accuracy underscores the GENR HDFLO model’s advanced capability in handling multimodal sentiment analysis, offering a more reliable and effective understanding of emotions in the data.

**Table 7. Error rate, and F1-score analysis of the proposed model over existing models.**
Techniques	MAE	F1-score (%)	Accuracy (%)
HyCon³⁷	0.71	85.1	85.2
AdaMoW³⁸	0.69	86.57	86.57
GENR HDFLO [proposed]	0.451	99.42	99.65

Figure 13 compares the performance of the proposed GENR HDFLO model against other models (ABCDM, DCN, PWDTT, PCA) based on different missing rates. In Fig. 13(a), the accuracy of each model decreases as the missing rate increases, but the GENR HDFLO model consistently achieves higher accuracy across all missing rates, demonstrating its robustness in handling incomplete data. In Fig. 13(b), the MAE increases with higher missing rates for all models. However, the GENR HDFLO model maintains a lower MAE compared to others, indicating its superior capability in minimizing prediction errors even with substantial data loss. These results underscore the effectiveness and reliability of the GENR HDFLO model in managing and analyzing multimodal data with missing values. The model excels in this area due to several key factors: it employs DSMOTE and ECGAN for effective data augmentation, which ensures a balanced dataset and minimizes issues such as noisy sample creation. Additionally, GENR HDFLO optimizes feature selection by considering the interdependencies among features from different modalities. MWCNN further enhances emotion classification by integrating spatial and spectral information, allowing it to capture complex data patterns more effectively. Together, these techniques result in higher accuracy and lower error rates compared to existing methods.

Fig. 13. Performance comparison of proposed model based on missing rates (a) accuracy (b) MAE.

4.11. Discussions

The proposed framework for MSA demonstrates significant advantages in effectively integrating audio, visual, and text modalities to understand emotions across diverse data sources. It exhibits superior performance compared to existing methods, achieving higher accuracy, recall, and precision in emotion classification tasks, particularly with the use of GENR HDFLO-based feature selection.

To demonstrate the novelty of the proposed work, this research conducted a comparative analysis with the existing models.^35,36 Table 8 presents a detailed comparison highlighting the advancements of the proposed model over the existing ones.

**Table 8. Shows the comparison analysis of the proposed model over existing models.**
		Input modality
Reference	Technique	Audio	Video	Text	Accuracy (%)
35	SVM	×	×	$✓$	91
36	BiGRU	×	×	$✓$	91
—	GENR HDFLO	$✓$	$✓$	$✓$	94.76

The GENR HDFLO model demonstrates advanced capabilities by integrating audio, video, and text modalities to achieve a high accuracy of 94.76%, surpassing existing SVM³⁵ and BiGRU³⁶ classifiers. Unlike SVM and BiGRU, which process only textual data, GENR HDFLO harnesses rich information from multiple modalities, enhancing its sentiment analysis capabilities. Existing models like SVM and BiGRU are limited by their reliance on single-modal data, which may hinder their ability to capture essential sentiment across different modalities. Additionally, single-modal classifiers may struggle with the inherent complexity and variability present in multimodal data, potentially resulting in lower accuracy compared to the proposed approach GENR HDFLO that integrates and interprets multiple modalities simultaneously. Therefore, the proposed model represents a significant advancement in MSA, addressing the limitations of existing single-modal approaches.

The proposed method, GENR HDFLO, offers several distinct advantages over existing methods in handling multimodal data with missing values. Firstly, it integrates DSMOTE and ECGAN for data augmentation, which effectively addresses class imbalance and ensures a more robust dataset without introducing noisy samples or overlapping classes. This approach leads to improved generalization and accuracy in classification tasks compared to traditional augmentation methods. Secondly, the use of GENR HDFLO for feature selection enhances the model’s ability to identify and incorporate the most relevant features from different modalities. This not only improves prediction accuracy but also ensures a more comprehensive representation of complex data relationships. By considering interdependencies among features across modalities, the model captures different patterns that may be missed by methods focusing solely on individual modalities.

However, the framework may suffer from high computational complexity, especially during feature selection and optimization stages. Future research will focus on optimizing model parameters to strike a better balance between performance and complexity, enhancing the framework’s scalability and efficiency. Additionally, efforts are needed to enhance the model’s robustness in handling missing modal information, ensuring its effectiveness in practical applications, and improving the interpretability of inter-modal interactions for better user understanding.

5. Conclusion

In this research, the MuSe-CaR dataset is introduced and collected from user-generated audio-visual and transcript recordings, which present imbalanced classes for testing and training. To address this, a data augmentation approach is employed to balance the three classes of multimodal data emotions. Various multimodal features were deeply represented with GENR which is optimized to minimize the loss function with HDFLO to select relevant features. These features are then utilized to classify emotions as valance, arousal, and trustworthiness using MWCNN, achieving high performance compared to existing methods with 0.6245 valences, 0.723 arousals, and 0.695 trustworthiness on MuSe-CaR. Notably, the feature selection method achieved 99.65% accuracy, 99.45% recall, and 99.66% precision, outperforming the existing deep learning and machine learning classifiers. Despite these achievements, future research efforts should prioritize (1) reducing model parameters while maintaining performance, (2) decreasing computational complexity, and (3) mitigating redundant information during model training. However, a notable challenge lies in enhancing the model’s robustness and accuracy in handling missing modal information, particularly in real-world scenarios. Addressing these challenges will expand the model’s applicability and reliability in practical settings involving MSA.

Acknowledgement

None

Data Availability Statements

Data sharing not applicable to this article as no datasets were generated or analysed during the current study.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

References

1. J. Wang, B. Xu and Y. Zu, Deep learning for aspect-based sentiment analysis, 2021 International Conference on Machine Learning and Intelligent Systems Engineering (MLISE) (2021), https://doi.org/10.1109/mlise54096.2021.00056. Crossref, Google Scholar
2. I. Priyadarshini and C. Cotton, A novel LSTM–CNN–grid search-based deep neural network for sentiment analysis, J. Supercomput. 77 (2021) 13911–13932, https://doi.org/10.1007/s11227-021-03838-w. Crossref, Web of Science, Google Scholar
3. H. Zhao, Z. Liu, X. Yao and Q. Yang, A machine learning-based sentiment analysis of online product reviews with a novel term weighting and feature selection approach, Info. Process. Manage. 58 (2021) 102656, https://doi.org/10.1016/j.ipm.2021.102656. Crossref, Web of Science, Google Scholar
4. S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu and J. Gao, Deep learning–based text classification, ACM Comput. Surveys. 54 (2021) 1–40, https://doi.org/10.1145/3439726. Crossref, Web of Science, Google Scholar
5. M. E. Basiri, S. Nemati, M. Abdar, S. Asadi and U. R. Acharrya, A novel fusion-based deep learning model for sentiment analysis of COVID-19 tweets, Knowl.-Based Syst. 228 (2021) 107242, https://doi.org/10.1016/j.knosys.2021.107242. Crossref, Web of Science, Google Scholar
6. A. Onan, Sentiment analysis on Massive open online course evaluations: A text mining and deep learning approach, Comput. Appl. Eng. Education. 29 (2020) 572–589, https://doi.org/10.1002/cae.22253. Crossref, Web of Science, Google Scholar
7. P. P. A., Performance evaluation and comparison using Deep Learning techniques in sentiment analysis, 3 (2021) 123–134, https://doi.org/10.36548/jscp.2021.2.006. Google Scholar
8. A. Onan, Sentiment analysis on product reviews based on weighted word embeddings and deep neural networks, Concurr. Comput.: Practice Experience. 33 (2020), https://doi.org/10.1002/cpe.5909. Web of Science, Google Scholar
9. C. Singh, T. Imam, S. Wibowo and S. Grandhi, A deep learning approach for sentiment analysis of COVID-19 reviews, Appl. Sci. 12 (2022) 3709, https://doi.org/10.3390/app12083709. Crossref, Google Scholar
10. R. Qasim, W. H. Bangyal, M. Alqarni and A. A. Almazroi, A fine-tuned BERT-based transfer learning approach for text classification, J. Healthcare Eng. 2022 (2022) 1–17. Crossref, Web of Science, Google Scholar
11. W. H. Bangyal, R. Qasim, N. U. Rehman, Z. Ahmad, H. S. Dar, L. Rukhsar, Z. Aman and J. Ahmad, Detection of fake news text classification on COVID-19 using deep learning approaches, Comput. Mathematical Methods Med. 2021 (2021) 1–14. Crossref, Web of Science, Google Scholar
12. Z. Kastrati, F. Dalipi, A. S. Imran, K. Pireva Nuci and M. A. Wani, Sentiment analysis of students’ feedback with NLP and deep learning: A systematic mapping study, Applied Sciences. 11 (2021) 3986, https://doi.org/10.3390/app11093986. Crossref, Google Scholar
13. E. Elgeldawi, A. Sayed, A. R. Galal and A. M. Zaki, Hyperparameter tuning for machine learning algorithms used for Arabic sentiment analysis, Informatics. 8 (2021) 79, https://doi.org/10.3390/informatics8040079. Crossref, Google Scholar
14. C. P. Cyril, J. R. Beulah, N. Subramani, P. Mohan, A. Harshavardhan and D. Sivabalaselvamani, An automated learning model for sentiment analysis and data classification of Twitter data using balanced CA-SVM, Concurrent Engineering. 29 (2021) 386–395, https://doi.org/10.1177/1063293x211031485. Crossref, Web of Science, Google Scholar
15. L. Stappen, A. Baird, L. Schumann and S. Björn, The Multimodal Sentiment Analysis in Car Reviews (MUSE-CAR) Dataset: Collection, Insights and Improvements, IEEE Trans. Affective Comput. 14 (2023) 1334–1350. Crossref, Web of Science, Google Scholar
16. L. Sun, M. Xu, Z. Lian, B. Liu, J. Tao, M. Wang et al., Multimodal emotion recognition and sentiment analysis via attention enhanced recurrent model, Proceedings of the 2nd on Multimodal Sentiment Analysis Challenge (2021), https://doi.org/10.1145/3475957.3484456. Crossref, Google Scholar
17. L. Stappen, L. Schumann, B. Sertolli, A. Baird, B. Weigell, E. Cambria and B. W. Schuller, MuSe-Toolbox: The multimodal sentiment analysis continuous annotation fusion and discrete class transformation toolbox, Multimodal Sentiment Analysis Challenge (2021). Crossref, Google Scholar
18. L. Stappen, A. Baird, E. Cambria and B. W. Schuller, Sentiment analysis and topic recognition in video transcriptions, IEEE Intell. Syst. 36 (2021) 88–95, https://doi.org/10.1109/mis.2021.3062200. Crossref, Google Scholar
19. B. Vlasenko, R. Prasad and M. Magimai-Doss, Fusion of acoustic and linguistic information using supervised autoencoder for improved emotion recognition, Multimodal Sentiment Analysis Challenge (2021). Crossref, Google Scholar
20. L. Stappen, A. Baird, M. Lienhart, A. Bätz and B. Schuller, An estimation of online video user engagement from features of time- and value-continuous, dimensional emotions, Frontiers in Computer Science 4 (2022), https://doi.org/10.3389/fcomp.2022.773154. Crossref, Google Scholar
21. L. Stappen, A. Baird, L. Christ, L. Schumann, B. Sertolli, E.-M. Meßner et al., The muse 2021 multimodal sentiment analysis challenge, Proceedings of the 2nd on Multimodal Sentiment Analysis Challenge (2021), https://doi.org/10.1145/3475957.3484450. Crossref, Google Scholar
22. D. Jiang, R. Wei, H. Liu, J. Wen, G. Tu, L. Zheng et al., A multitask learning framework for multimodal sentiment analysis, 2021 International Conference on Data Mining Workshops (ICDMW) (2021), https://doi.org/10.1109/icdmw53433.2021.00025. Crossref, Google Scholar
23. A. Yadav and D. K. Vishwakarma, A deep multi-level attentive network for multimodal sentiment analysis, ACM Trans. Multimed. Comput. Commun. Appl. 19 (2023) 1–19. Crossref, Web of Science, Google Scholar
24. V. S. J. Padminivalli and M. C. Rao, Deep aspect-sentinet: Aspect based emotional sentiment analysis using hybrid attention deep learning assisted BILSTM, International J. Uncertainty Fuzziness Knowl. Based Syst. 32 (2024) 21–51. Link, Web of Science, Google Scholar
25. L. He, Y. Chen, C. Zhong and K. Wu, Granular elastic network regression with stochastic gradient descent, Math. 10 (2022) 2628, https://doi.org/10.3390/math10152628. Crossref, Google Scholar
26. E. Odhiambo Omuya, G. Onyango Okeyo and M. Waema Kimwele, Feature selection for classification using principal component analysis and information gain, Expert Syst. Appl. 174 (2021) 114765, https://doi.org/10.1016/j.eswa.2021.114765. Crossref, Google Scholar
27. H. Zhou, J. Zhang, Y. Zhou, X. Guo and Y. Ma, A feature selection algorithm of decision tree based on feature weight, Expert Syst. Appl. 164 (2021) 113842, https://doi.org/10.1016/j.eswa.2020.113842. Crossref, Web of Science, Google Scholar
28. R. K. Behera, M. Jena, S. K. Rath and S. Misra, Co-LSTM: Convolutional LSTM model for sentiment analysis in Social Big Data, Info. Process. Manage. 58 (2021) 102435, https://doi.org/10.1016/j.ipm.2020.102435. Crossref, Web of Science, Google Scholar
29. M. E. Basiri, S. Nemati, M. Abdar, E. Cambria and U. R. Acharya, ABCDM: An attention-based bidirectional CNN-RNN deep model for sentiment analysis, Future Gener. Comput. Syst. 115 (2021) 279–294, https://doi.org/10.1016/j.future.2020.08.005. Crossref, Web of Science, Google Scholar
30. Z. Liu, B. Zhou, D. Chu, Y. Sun and L. Meng, Modality translation-based multimodal sentiment analysis under uncertain missing modalities, Info. Fusion 101 (2024) 101973, https://doi.org/10.1016/j.inffus.2023.101973. Crossref, Google Scholar
31. G. Chandrasekaran, S. Dhanasekaran, C. Moorthy and A. A. Oli, Multimodal sentiment analysis leveraging the strength of deep neural networks enhanced by the XGBoost classifier, Comput. Methods Biomech. Biomed. Eng. (2024) 1–23, https://doi.org/10.1080/10255842.2024.2313066. Crossref, Web of Science, Google Scholar
32. P. Wang, S. Liu and J. Chen, CCDA: A novel method to explore the cross-correlation in dual-attention for multimodal sentiment analysis, Appl. Sci. 14 (2024) 1934, https://doi.org/10.3390/app14051934. Crossref, Google Scholar
33. S. J. R. K. V. Padminivalli and M. V. P. C. S. Rao, Deep Aspect-Sentinet: Aspect based emotional sentiment analysis using hybrid Attention Deep Learning Assisted BILSTM, International J. Uncertainty, Fuzziness Knowledge-Based Syst. 32 (2024) 21–51, https://doi.org/10.1142/s0218488524500028. Link, Web of Science, Google Scholar
34. K. Anuratha and M. Parvathy, Twitter sentiment analysis using social-spider LEX feature-based syntactic-senti rule recurrent neural network classification, International J. Uncertainty, Fuzziness Knowledge-Based Syst. 30 (2022) 45–66, https://doi.org/10.1142/s0218488522400037. Link, Web of Science, Google Scholar
35. W. H. Bangyal, M. Iqbal, A. Bashir and G. Ubakanma, Polarity classification of twitter data using machine learning approach, in 2023 International Conference Human-Centered Cognitive Syst (HCCS) (2023) 1–6, https://doi.org/10.1109/hccs59561.2023.10452557. Crossref, Google Scholar
36. W. H. Bangyal, S. Amina, R. Shakir, G. Ubakanma and M. Iqbal, Using deep learning models for COVID-19 related sentiment analysis on Twitter data, in 2023 International Conference Human-Centered Cognitive Syst (HCCS) (2023) 1–6, https://doi.org/10.1109/hccs59561.2023.10452567. Crossref, Google Scholar
37. S. Mai, Y. Zeng, S. Zheng and H. Hu, Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis, IEEE Transactions on Affective Computing 14 (2022) (3) 2276–89. Crossref, Google Scholar
38. J. Zhang, X. Wu and C. Huang, AdaMoW: Multimodal sentiment analysis based on adaptive modality-specific weight fusion network, IEEE Access 11 (2023) 48410–20. Crossref, Web of Science, Google Scholar
39. X. Zhou and W. Li, Local convolutional neural network based pop-up text recognition and sentiment analysis. International J Uncertainty Fuzziness Knowledge-Based Syst 32 (2024) 409–31. Link, Web of Science, Google Scholar
40. Z. Wang, S. B. Ho and E. Cambria, Multi-level fine-scaled sentiment sensing with ambivalence handling. International J Uncertainty Fuzziness Knowledge-Based Syst 28 (2020) 683–97. Link, Web of Science, Google Scholar