1. Introduction[1]

Humans communicate with each other both verbally and non-verbally. Verbal communication involves the use of sentences and words during interaction, while non-verbal communication includes activities such as head and body movements, along with changes in speech tone, all of which develop and complement the meaning expressed through words. Research indicates that over 90% of concepts exchanged between people are communicated through non-verbal channels, with facial expressions, tone of speech, and body movements playing a significant role among them [1]. Emotion is a vital aspect of non-verbal communication. For instance, frowning before speaking often signals dissatisfaction or disagreement. Emotions such as surprise, happiness, and fear are natural human reactions to environmental factors [2]. Moreover, the unconscious expression of some emotions plays an important and guiding role in human relationships.

The use of computers in programs that interact with users relies on accurately identifying their moods and emotions. Computers must be able to recognize emotions conveyed through speech tone, facial expressions, or body movements. Human movement and interaction are inherently intertwined with media. The senses of sight and hearing are essential for navigating our complex world. Numerous studies in psychology have explored how to assist people in completing tasks by combining auditory and visual cues, such as lip reading [3] and sound source localization [4]. These studies provide a foundation for research into intelligent systems that integrate auditory and visual modalities. It can be said that humans are sophisticated systems equipped with multimodal sensors that can process, analyze, learn from, and respond to various stimuli [5-7].

Considering the importance of information fusion, this article leverages audio and visual information fusion to recognize emotional states. One of the most significant challenges in integrating speech and facial features lies in managing the interaction between the temporal and spatial dimensions of audio-visual data and ensuring synergetic processing of their nonlinear relationships. To address this, a new model for multimodal emotion recognition is presented in this article. The proposed framework employs time-spatial sequences to learn the intricate relationships between frames and features of facial emotional states, as well as emotional speech, thereby effectively integrating the audio-visual processes.

This article introduces an innovative enhancement in the image segmentation stage of the pipeline by applying Grey Wolf Optimization (GWO). Inspired by the natural hunting behavior of grey wolves, GWO is used to adaptively segment the extracted images into meaningful regions. This segmentation process enables the extraction of more discriminative features from facial data, which contributes to a higher recognition rate in the subsequent layers of the model.

After segmentation, the images and speech signals undergo preprocessing, followed by feature extraction using deep learning techniques. The framework employs a two-channel feature structure, where CNN-based feature extraction identifies key discriminative attributes in both modalities. The process begins with convolution and pooling operations, progressively refining and reducing the dimensionality of the features.

A unique fusion layer combines these features through convolutional fusion, ensuring effective dimensionality reduction before sending them to the Softmax layer and self-attention modules. Finally, the features are concatenated in a cascade fashion, merging optimal features from both modalities for further processing.

The classification section employs a fully connected layer that utilizes reinforcement learning-based automata learning [C1] to optimize the model’s parameters and enhance classification performance. This approach allows the proposed model to effectively compress high-dimensional data while preserving essential and distinctive features of the inputs, ultimately improving overall recognition accuracy. By integrating Grey Wolf Optimizer (GWO) for image segmentation and automata learning for adaptive training in the final layer, the model demonstrates significant advancements in multimodal emotion recognition.

The structure of this article is organized such that Section 2 reviews existing works on the recognition of emotional states in facial expressions, speech, and their fusion. Section 3 will detail the proposed method, while Sections 4 and 5 will present the results of the experiments and the conclusions, respectively.

2. Recognition of Emotional States

2.1. Face Emotion Recognition

Expressing emotions plays a crucial role in everyday communication, and one of the key methods for detecting these emotions is recognizing facial expressions. Consequently, an important aspect of natural human-machine interaction is the ability to recognize facial expressions and provide feedback based on the perceived emotions. These facial changes result from muscle movements in various areas of the face. To explore this topic, extensive research has been conducted on facial expressions and muscle activity, including the development of the Facial Action Coding System (FACS) [8]. In [9], the relationship between facial expression recognition and mental characteristics is examined. Based on this analysis, a facial expression recognition model is developed that incorporates convolutional neural network (CNN) layers and integrates various neural network architectures such as VGGNet, AlexNet, and LeNet-5. Additionally, a Central Local Binary Pattern (CLBP) algorithm optimized for CNN is introduced, resulting in the creation of a CNN-CLBP algorithm for facial expression recognition.

Finally, a validity analysis is conducted on this algorithm following the preprocessing of facial images and optimization of key parameters. Compared to other methods, the CNN-CLBP algorithm achieves a higher accuracy in recognizing facial expressions, with an average recognition rate of 88.16%. Furthermore, the accuracy of this algorithm is improved through image preprocessing and parameter optimization, leading to a robust performance. Notably, the CNN-CLBP algorithm can accurately identify 97% of happy expressions and surprised expressions; however, it has a misidentification rate of 22.54% for sad expressions. In [10], a face recognition system is developed based on a multi-scale feature hybrid architecture, aimed at fully utilizing facial characteristics to improve recognition accuracy. Additionally, three different scaling methods are designed to extract the global features of faces. The multi-layer bilinear features from various scales are integrated through a multi-level bilinear integration layer. This approach captures several feature relationships across different levels, enhancing the model’s ability to extract and identify subtle facial features. Furthermore, it suggests using layer-by-layer convolution to combine multi-layer feature data, addressing the challenge of losing key features while extracting from multi-layer convolution and integration layers. The experiments demonstrate that the recognition accuracy of this method, applied to the Yale, AR, and ORL face databases, significantly surpasses that of traditional algorithms. In [11], the researchers made significant advancements in convolutional neural network (CNN) techniques to identify seven fundamental emotions. They also evaluated several preprocessing methods to determine how these factors impact CNN performance. This study focuses on enhancing facial features and expressions related to emotion recognition. By detecting facial expressions that elicit human responses, computers can more accurately predict individuals' mental states and provide more personalized reactions. In [12], a novel hybrid neural network (NN) is proposed, capable of detecting micro-expressions in real-time applications. This study compares several NN models for the first time. Subsequently, a hybrid NN is developed by combining a CNN, a recurrent neural network (RNN) known as long short-term memory (LSTM), and a vision transformer. In [13], the authors introduced a method for facial emotion recognition in concealed facial images using low-light image enhancement and feature analysis of upper facial characteristics with a CNN.  In [14], two primary steps are proposed for implementing the suggested method. The first step involves reading a video, converting it into images, and preprocessing those images. The second step utilizes a combination of a 3D convolutional neural network (3DCNN) and learning automata (LA) to classify and detect the emotional recognition rate of the face.

• Speech Emotion Recognition

Discourse flags are a quick and common way for individuals to communicate. This reality has led researchers to explore the use of discourse as an efficient communication method between machines and people [4, 15]. Recently, significant research has been conducted in the field of discourse flag processing and classification to identify emotional discourse using machine learning models. Within the article [16], Convolutional Neural Networks (CNNs) are employed to extract features from discourse. The initial layer of the CNN is utilized for discourse recognition, focusing on extracting emotional cues. In [17], which utilizes Deep Convolutional Neural Networks (DCNN), AlexNet has been applied for emotional discourse recognition. AlexNet is trained using the large-scale ImageNet database [18]. The approach demonstrated a reduction in preparation time while accommodating a wide variety of question classes. The structure utilized AlexNet, which includes an input layer, convolutional layers, integration layers, and fully connected layers (FCL). In the article [19], the fourth convolution layer (Conv4) was employed to extract low-level features. In [20], significant contributions are highlighted for (i) improving the accuracy of speech emotion recognition (SER) compared to the current state of the art, and (ii) reducing the computational complexity of the proposed SER model. The authors introduce a deep step convolutional neural network (DSCNN) assisted by artificial intelligence, which uses a straightforward systematic approach to learn both general and specific features from speech signal spectrograms processed in earlier stages for enhanced performance. Despite the promising advancements in recent studies, advanced combination algorithms are generally not utilized to produce effective representations of emotional features in speech. To address this issue, reference [21] describes the integration of spatial and temporal feature representations of speech emotions using a parallelized Convolutional Neural Network (CNN) and a Transformer encoder for Speech Emotion Recognition (SER). The authors stack two parallel CNNs to capture spatial features alongside a Transformer encoder that represents temporal features. This approach increases the channel depth while reducing the feature dimension through expressive hierarchical feature representation, all with lower computational cost. To tackle these challenges, reference [22] introduces a new SER system called IMEMD-CRNN, which combines an improved version of envelope signal-based Empirical Mode Decomposition (IMEMD) with a Convolutional Recurrent Neural Network (CRNN). First, IMEMD is proposed for speech parsing. It employs a perturbation-assisted EMD technique and can determine the parameters of the envelope signals based on the characteristics of the signals. Next, the system extracts 43-dimensional time-frequency features that characterize emotions from the intrinsic mode functions (IMFs) obtained through IMEMD. These features are then input into a CRNN model for emotion detection. In the CRNN, two-dimensional CNN layers are used to capture spatial, temporal, and frequency-related nonlinear data from emotional speech. Bidirectional Gated Recurrent Units (BiGRU) are utilized to enhance the learning of temporal contextual information. In reference [23], an efficient method for detecting emotional states from speech signals is proposed, based on a new cognitive model called SOAR. This paper emphasizes the significance of the topic and introduces a hybrid deep learning and cognitive model that leverages CNNs. This approach ensures that no dimensions are discarded from the speech signal, resulting in more effective classification by considering temporal information in dynamic speech. Additionally, the training of the CNN network incorporates Learning Adjustment (LA) to improve the efficiency of the proposed model, and the working memory component of the SOAR model can be applied.

2.3. Emotion Recognition Using Information Fusion

Data combination refers to a set of techniques that utilize information from multiple sources to obtain increasingly accurate insights about a subject. In [24], six basic emotional expressions are considered: anger, disgust, fear, happiness, sadness, and surprise. The proposed model consists of four main components: preprocessing, feature extraction, feature selection, and classification. During the preprocessing stage, facial images and verbal signals are extracted from videos taken from the Enterface’05 dataset. This process involves denoising and resizing the data. In the feature extraction stage, Principal Component Analysis (PCA) is applied to the images, while a 3D Convolutional Neural Network (CNN) is used to identify the best features within these images. Additionally, Mel-frequency cepstral coefficients (MFCC) are applied to the emotional verbal signals, and another CNN is employed to extract optimal features from them. Afterward, the fusion of the extracted features is performed. Finally, the fused features are analyzed using Soar classification to calculate the recognition rate of emotional expressions based on both facial cues and speech. The need for such techniques is evident in various ways. In classification tasks, depending on the structure of the problem and the chosen arrangement, data combination can occur at four levels: data, features, classification, and decision-making. If different sensors measure similar phenomena (for example, observing an object from different angles or combining primary colors), the fusion process takes place at the data level. It is also possible to extend the fusion process to the feature level. Many psychological studies, both theoretically and empirically, have demonstrated the importance of integrating information from multiple modalities—such as emotional speech and facial expressions—to create a coherent representation and understanding of emotions [25]. As a result, numerous studies have been conducted on human auditory-visual perception recognition in recent years [26, 27]. Additionally, research has been carried out in other areas, such as the combination of facial expressions and head movement data [28], as well as the combination of facial expressions and body movement or behavior [29]. Most of these approaches do not rely on speech content. When different sensors measure similar values of a phenomenon (for example, observing an object from different angles or combining primary colors), the integration process can occur at the data level. It is also possible to perform the combination at the feature level. Various psychological studies, both theoretical and experimental, have demonstrated the significance of integrating information from multiple modalities. For instance, in [30], a biometric method based on the combination of finger and facial vein features is proposed using convolutional neural networks (CNNs). In this approach, integration takes place within the feature layer. The self-attention mechanism is employed to obtain the biometric weights, which are then combined with the remaining structure of RESNET. The self-attention weighted features are cascaded with the concatenated bimodal integration feature channel. In [31], a novel multi-layer interactive highlight crossover organization model with a precise separation loss is proposed. First, a multi-layered and multi-scale module is designed to extract global and local features of facial expressions, capturing various relationships between different scales. This enhances the model's ability to detect subtle features. Next, a hierarchical interactive feature fusion module, called Moment Face Emotions, is implemented to address the loss of valuable feature information that results from the complexity of layer-by-layer processing and integration in CNNs. Additionally, an attention mechanism is employed between the convolutional layers at different levels. This improves the neural network's detection capability by increasing the prominence of relevant information about different features at various layers while suppressing irrelevant data. Finally, we use the precise separation loss function to enhance the inter-class feature separation and intra-class feature clustering abilities of the proposed model, thereby addressing issues related to high intra-class variability and low inter-class similarity in facial expression recognition. In [32], the creators proposed a modern multimodal combination neural network (MFFNN) model that effectively extracts complementary information from eye movement signals and performs feature fusion with EEG signals. We developed a two-branch feature extraction module to obtain features from both modalities while ensuring temporal alignment. A multi-scale feature fusion module is introduced, which utilizes cross-channel soft attention to adaptively select information from different spatial scales, allowing for the acquisition of features at various spatial resolutions for effective integration. In [33], multimodal emotion recognition is introduced to identify facial expressions, speech intonations, and behaviors in the context of adaptive higher education management. Language and expression are the primary means through which individuals convey their emotions. After implementing a dual measurement detection system framework, the Bag of Words (BOW) model is used to identify key features and regions of interest. In [34], a multimodal interaction-based representation learning system is proposed for recognizing feelings from a confrontational video. This approach begins with a semantic enhancement module designed to guide the audio/visual encoder using semantic data from the text. Then, a multimodal bottleneck transformer is utilized to improve audio and visual representations by modeling dynamic cross-modal interactions between the two feature sequences. Test results on two databases demonstrate the superiority of our proposed method. With semantically enriched audio and visual features, it outperforms state-of-the-art models that combine features or decisions from audio, visual, and textual modalities. In [35], a study explores the methods and tools used to identify emotions in children. This research presents a literature review employing a systematic approach and the PRISMA method to report both quantitative and qualitative findings. Various perception channels and techniques are analyzed, including facial expressions, speech prosody, and physiological signals. According to representational models, fundamental emotions—particularly joy, fear, and sadness—are better understood and recognized. In [36], a progressive flag handling strategy utilizing deep neural networks (DNN) for emotion recognition based on EEG signals is presented. The spectral and temporal components of the raw EEG signal are first preserved in a 2D spectrogram before feature extraction. A pre-trained AlexNet model is employed to extract raw features from the 2D spectrum for each channel. To reduce the dimensionality of the spatial and temporal features, a Bag of Deep Features (BoDF) model is proposed. This model calculates a set of words consisting of 10 cluster centers from each class using the k-means clustering algorithm. Ultimately, the representation for each subject is constructed using a histogram of the vocabulary set derived from the raw features of a channel. The features extracted from the proposed BoDF model have significantly smaller dimensions. In [37], a Deep Normalized Attention-Based Remaining Convolutional Neural Network (DNA-RCNN) is proposed to extract relevant features based on the discriminative representation of features. This neural network also explores attractive features using proposed attention modules that lead to consistent performance. Classification is then performed using the proposed Modified Random Forest (M-RF) with an observational loss function. In this process, learning weights are assigned to the subset of data to minimize the loss between the predicted values and the ground truth, contributing to accurate classification.

In this article, we have utilized deep learning techniques derived from Convolutional Neural Networks (CNNs). The advantage of using CNNs is that their architectures can be fine-tuned to achieve better and more accurate results. The more optimized the network is and the more data it is trained on, the better its performance will be compared to traditional machine learning algorithms. Additionally, in a CNN, each neuron is not connected to all the neurons in the previous layer, but only to a limited number of them. This selective connectivity helps reduce the number of parameters in the model. Furthermore, certain connections can share the same weight, which also contributes to parameter reduction. The pooling layer is particularly beneficial as it decreases the volume of data processed by preserving only the essential information, making CNNs suitable for large datasets. While CNNs offer many advantages, the calculations involved in the max-pooling layer and the fully connected layer can slow down the overall process. Therefore, we have implemented several modifications in the final layer of the proposed model.

Fig. 1. Block Diagram of the Proposed Method

3. The Proposed Model

In this article, we explore the use of two inputs: speech emotional signals and images of emotional facial expressions. Figure 1 illustrates the block diagram of the proposed method. The model incorporates two inputs:

Path of speech signals: one-dimensional discourse signals are changed over into improved spectrogram pictures. At that point, the produced RGB pictures are sent to the pre-trained AlexNet model to be tuned for passionate discourse acknowledgment. In encourage clarification, it can be said that the improved spectrogram will work based on the log Mel spectrogram, which is a successful strategy for analyzing the one-dimensional discourse flag based on its recurrence in brief interims and showing it as a picture. The length of the outlines will have a critical effect on the quality of the resulting picture. Based on this and in this article, the recurrence determination based on the testing recurrence of the sound signals utilized is set to 16 kHz. The arrangement of FFT is set to 1024, and the range pictures are 512 pixels wide. Since we need to take advantage of all the data given within the spectrographs, we consider three color channels, R, G, and B. The improved spectrograph is made based on three spectrographs obtained by three distinctive time outlines of 8, 16, and 32 milliseconds. In the first layer of the proposed model, all inputs are sized at 277 x 227 x 3 and should be resized for compatibility with the dimensions of the speech signal in the input layer. The convolutional layer (CL) consists of convolutional filters that are used to extract various local patterns from each region in the input, generating multiple feature maps. The AlexNet model contains five convolutional layers, labeled Conv1, Conv2, Conv3, Conv4, and Conv5. Of these, Conv1, Conv2, and Conv5 are followed by max-pooling layers. The convolutional layers utilize the ReLU activation function to enhance the learning process. The first convolutional layer, Conv1, contains 96 kernels with dimensions of 3 x 11 x 11, a stride of 4 pixels, and applies zero-padding. The second layer, Conv2, has 256 filters with a size of 48 x 5 x 5, a stride of 1, and a padding of 2. Conv3, the third convolutional layer, features 384 filters with dimensions of 3 x 3 x 256, which are connected to the outputs of Conv2. The fourth convolutional layer, Conv4, consists of 384 filters with dimensions of 3 x 3 x 192. The ReLU activation function is applied at the output of each convolutional layer to facilitate the training process.

Face images path: The face images, which are multi-dimensional and in RGB format, are sent to the pre-trained AlexNet model for tuning in emotional face recognition. In this study, the video file's sampling frames are set to 10 frames. All inputs in the first layer have dimensions of 255 x 255 x 3. The convolutional layer (CL) consists of convolutional filters used to extract various local patterns from specific areas in the input, generating several feature maps. The AlexNet model comprises five convolutional layers: Conv1, Conv2, Conv3, Conv4, and Conv5. Among these, three layers (Conv1, Conv2, and Conv5) are followed by max-pooling operations. The convolutional layers utilize the ReLU activation function. The first convolutional layer (Conv1) contains 64 cores, with a size of 3 x 8 x 8, a stride of 4 pixels, and zero-padding. The second layer (Conv2) has 256 cores and a size of 64 x 3 x 3, with a stride of 1 and padding of 2. The third convolutional layer (Conv3) consists of 384 cores with a size of 128 x 3 x 3 and is connected to the outputs of Conv2. Lastly, the fourth convolutional layer (Conv4) also has 384 cores, with a size of 164 x 3 x 3. The ReLU activation function is applied to adapt to the training process at the output of each convolutional layer.

The highlights extracted from two channels within the convolutional layer (Conv4) will be processed using a feature selection method to identify the most significant features. To implement the semantic memory portion, we utilize the Correlation-based Feature Selection (CFS) method, which selects distinct features. The CFS technique evaluates a subset of features and chooses only those that have a high correlation with the output class label. This classification algorithm, based on these selected features, improves predictive accuracy. In the continuation of the feature combination module, convolutional combination is first used to reduce the dimensionality, and the resulting features are then sent to the Softmax layer, which obtains the self-attention weights. The most important features are subsequently passed to the concatenation channel to merge the two sets of features. During the convolutional combination step, the parameters of each convolution unit are optimized using backpropagation algorithms informed by learning automata (LA).

The following section provides further explanations to evaluate the proposed model:

Feature extraction is performed using AlexNet. Within AlexNet, most of the network's weights remain fixed while several layers are utilized for the feature extraction process. AlexNet features a deep architecture with multiple channels per layer and stacked convolutional layers. These convolutional layers include max pooling, dropout, activation functions like ReLU, data augmentation, and stochastic gradient descent with momentum. The model employs ReLU as its activation function, which accelerates the processing speed. In the fully connected layers (FCLs), the output of the network is represented as a vector of a specific size, which can be used for further processing. However, a drawback of these layers is their large number of parameters, which results in a high computational cost during training. For this reason, FCLs do not perform well in recognizing and analyzing raw data.

In contrast, the convolutional layers (specifically Conv4) extract important features from raw input values while maintaining spatial relationships within the data. FCLs capture general features, whereas convolutional layers (CLs) generate local features and transform local descriptors into a compact feature vector. In the proposed framework, the features extracted from the convolutional layer (Conv4) are utilized for sentiment detection. This local convolution layer (Conv4) enables the model to learn the channels with the strongest response in a specific local input region, thereby capturing the local relationships within the input space. A total of 64,896 features have been obtained from the convolutional layer (Conv4) and are sent to the feature selection phase. This article employs a correlation-based feature selection (CFS) method for feature selection. The CFS method assesses a subset of features and selects only those that have a strong correlation with the target outcome [43]. The CFS (Correlation-based Feature Selection) positions are highlighted, which will be calculated using an exploratory assessment based on relationships. This strategy has been employed to measure and evaluate the similarity between features. The CFS method eliminates irrelevant attributes that are less relevant to the class label. The CFS criterion is calculated using equation “(1)” [43]:

Where  is the classification correlation of the feature, k is the number of features, and  represents the correlation between the features. The extracted features are utilized alongside a Convolutional Neural Network (CNN) in an optimized feedforward mode. This image processing method effectively reduces the dimensions of large datasets while preserving the distinctive characteristics of the images, aligning with the principles of image processing. In the feature fusion module, convolutional fusion is employed to reduce the dimensions, and the parameters of each convolution unit are optimized using Learning Algorithm (LA) backpropagation techniques. The rationale behind using LA in neural network learning is that the optimization problem typically lacks convexity. Therefore, a common approach to solving optimization issues in neural networks is through backtracking. To improve the CNN, we incorporate the probability theory of the LA within the Backpropagation Process (BPP) errors to train the proposed model. When LA is applied in this layer, it not only reduces computational complexity but also increases the model's speed compared to traditional gradient descent methods. The output of the LA, denoted as \ (i - \alpha \), represents potential actions that serve as momentum factors for updating the neural network's weights. Through multiple iterations, we establish a strong connection between the optimal input and output features. This process allows the parameters to be adaptively adjusted in the gradient descent (GD) method for calculating backpropagation error. Subsequently, the output from the convolutional fusion (Fusion_Conv) is directed to the Softmax layer, where it acquires self-attention weights. The best features extracted are then channeled into a concatenated cascade to fuse the two feature sets. LA functions as a stochastic model within the framework of reinforcement learning, where automatic learning utilizes selected actions as inputs to operate within its stochastic environment. The environment provides reinforcement feedback to indicate how closely the adopted actions align with the desired outcomes, enabling the action probability vector to be updated. Automatic learning aims to identify the optimal action from a set of possible actions to minimize the average penalty associated with the environment. This approach is particularly useful in scenarios where complete information about the environment is lacking. The output from the LA, being the potential actions, iteratively updates the neural network's weights, thereby strengthening the connection between the most relevant input and output features and adjusting parameters accordingly.

4. Results and Experiments

4.1. Database

In our analysis of Ryerson's audio-visual database, we have utilized the RAVDESS[i] [38] emotional speech and song dataset. This dataset consists of 7,356 recordings (total size: 24.8 GB) featuring emotionally expressive speech, where two lexically matched phrases are voiced in a neutral North American accent. The recordings are evenly distributed across three content types: full audio-visual (AV), video-only, and audio-only. There are also two audio channels: speech and music. Each emotional expression is produced at two levels of intensity—normal and strong—with an additional neutral expression. All conditions are available in three formats: audio-only (16-bit, 48 kHz .wav), audio-video (720p H.264, AAC 48 kHz, .mp4), and video-only (no audio). It’s important to note that there is no song file for Actor 18. In addition to the neutral emotion, which only has a regular intensity, the other emotional expressions are produced at two levels of arousal: regular and strong. Each recording features an actor representing one of the following eight emotions: Calm (925), Neutral (892), Happy (796), Sad (986), Angry (921), Fear (907), Surprised (974), and Disgusted (955).

We used the entire dataset to propose a multimodal approach for our experiments, focusing on the recognition of audio-visual emotions in speech. The videos in the dataset have maximum and minimum durations of 5.31 seconds and 2.99 seconds, respectively. The ensemble comprises 24 actors (12 female and 12 male), ensuring a balanced representation of genders, who express lexically matched phrases in a neutral North American accent. This collection is well-suited for studying emotion-related metalinguistic features, lexical isolation, and reducing the bias that culture can introduce in emotional expressions. One key advantage of the dataset is its proportional number of files for each emotion, which helps avoid challenges related to training algorithms on unbalanced data. Furthermore, RAVDESS is considered a reference dataset within the research community and has been employed in several studies [23–39].

In our proposed method, we employed a hierarchical structure inspired by the social behavior and hunting processes of wolves to enhance feature extraction and image segmentation. This optimization process is guided by three key roles: alpha, beta, and delta wolves. The alpha wolf acts as the primary leader, directing the search process, while the beta and delta wolves help refine the search area. The remaining wolves serve as followers within the optimization hierarchy.

The Grey Wolf Optimization (GWO) algorithm is specifically used to improve the segmentation and feature extraction phases of the model. During these enhancements, GWO employs an evaluation function within its fitness module, which considers key parameters for optimization. In this equation, the weight represents the feature weighting factor, while  and  are randomly generated numbers between 0 and 1. The parameter M acts as a component weight that adjusts the influence of random searches across different search iterations. This structure effectively balances exploration and exploitation, helping to locate optimal solutions within the image segmentation space.

By integrating GWO, segmentation operations are applied to the extracted images in both the facial and speech feature channels. The segmented images create refined regions that facilitate the extraction of more discriminative features, thereby enhancing the overall recognition accuracy of the proposed model. The hierarchical segmentation, guided by the alpha, beta, and delta wolves, ensures precise identification of feature-rich areas in the images, enabling the model to achieve superior performance.

4.2. Experimental Results

By using the Fourier transform, all grayscale images are converted into the frequency domain. In the image processing stage, histogram matching is performed, followed by histogram equalization, which is applied to all three channels (Red, Green, and Blue) of the image. A gamma transform is also applied to each channel, using gamma values of 5 and 0.2. Additionally, gradient and Laplacian filters are utilized to enhance image clarity. We will then apply three different low-pass filters to the frequency domain representations of the images. Finally, we will obtain the RGB color image and the image histograms separately for the Hue, Saturation, and Intensity channels. An example of this operation is illustrated in Figure 2.

Fig. 2. Performing Various Types of Pre-Processing on Images

The log Mel spectrogram is an effective method for analyzing one-dimensional speech signals based on their recurring components over short time intervals, presenting this information visually. The duration of the time windows significantly impacts the quality of the resulting image. Short time windows generate wideband spectrograms with high and low spectral resolution, while longer windows yield lower resolution. All experiments were conducted using the Leave-One-Speaker-Out (LOSO) technique. Emotions are closely linked to the speaker's identity. To apply this algorithm effectively and learn the relationship between speech and emotion independently of the speaker's identity, it is essential to train the system using multiple speakers and test it with others who have no samples in common. For this reason, the RAVDEESS dataset was utilized, as it includes speech samples from ten different speakers. During the testing procedure, ten independent trials were conducted: in each trial, samples from nine speakers were used for training, while the samples from the remaining speaker were kept hidden for testing. This process continued until all ten speakers had participated in the testing phase. Additionally, to evaluate the proposed method, tests were performed on the dataset to determine how the integration of various features could enhance classification performance. Tables 1 to 3 present spectrograms of wideband (8 ms time window), midband (16 ms time window), and narrowband (32 ms time window), along with the characteristics of the images in data integration, specifically in terms of confusion matrices. The results, along with the obtained data, are displayed in the accompanying figures.

As shown in Table 1, the accuracy of the proposed method using the broadband spectrometer (with a response time of 8 milliseconds) has reached 86.47%. In terms of the recall criterion, the highest value was for the emotion of feeling relaxed, while the lowest was for feeling disgusted.

Table 2 presents the accuracy achieved by the proposed method using the medium band spectrograph (with a response time of 16 milliseconds), reaching 88.75%. In this instance, accuracy was highest for the emotion of feeling sad and lowest for feeling angry. Similar to the previous results, the recall criterion indicated that the highest value was for feeling relaxed, and the lowest was for feeling disgusted.

Table 3 illustrates the accuracy of the proposed method with the narrowband spectrograph (with a response time of 32 milliseconds), achieving an accuracy of 87.46%. In this experiment, the highest accuracy corresponded to the emotion of feeling sad, while the lowest was for feeling surprised. Once again, in the recall criterion, the highest value was for feeling relaxed, and the lowest was for feeling disgusted.

Table 4 presents the confusion matrix generated by the proposed method using the RAVDEESS dataset. In these tables, the columns represent the identified emotions, while the rows denote the actual emotions. The last column of the table shows the recall, defined as the number of accurately identified samples from each emotion class divided by the total number of samples in that class. This indicates the recognition rate for each category.

Accuracy can be calculated by determining the reliability of the classification results for each emotion class. This is done by dividing the number of correctly classified samples for each class by the total number of samples assigned to that class by the classifier. Additionally, overall accuracy or detection rate is evaluated as the total number of correctly classified samples divided by the overall number of samples in the dataset.

In the following discussion, we will compare the performance of ResNet 152, VGGNet 16, AlexNet, and LeNet 5 to determine the best CNN architecture to use within the proposed method.

Table (1): Perplexity matrix of wide-band spectrogram (8ms time frame) in RAVDEESS Dataset

Table (2): Perplexity matrix of wide-band spectrogram (16ms time frame) in RAVDEESS Dataset

Table (3): Perplexity matrix of wide-band spectrogram (32ms time frame) in RAVDEESS Dataset

Table (4): Perplexity matrix of wide-band spectrogram in RAVDEESS Dataset

Fig. 3. Comparison of VGGNet 16, ResNet 152, AlexNet, LeNet 5 on RAVDEESS dataset

As illustrated in Figure 3, the proposed method utilizing the ResNet152 network achieved an accuracy of 89.44%, making it superior to all other methods evaluated. Consequently, this approach is classified as the type of CNN employed in the proposed framework. In the next evaluation, we will compare and analyze the proposed method alongside similar techniques. In our proposed framework, three experiments were conducted on the input data. The first set of experiments focused on unimodal biometrics by analyzing speech and facial streams independently, employing feature extractors. In the second and third series of experiments, we assessed the fusion of these two biometrics—speech and facial streams. This fusion was first carried out using feature extractors and then without them, to evaluate the impact of feature extraction on overall performance. Table 5 presents a comparison between the method that utilized normal extractors and the one that benefited from segmentation improvement provided by GWO. According to Table 5, the proposed method that implemented segmentation yielded better results than the extraction method that did not use segmentation. In both methods, the emotion of disgust achieved the highest recognition rate, with the recognition accuracy percentage showing an improvement of 1.2% when segmentation was applied compared to the method without it.

As illustrated in Table 6, the proposed method achieves higher accuracy than the other investigated methods, thanks to the integration of new information and information fusion for classification.

Table (5): Comparison of recognition accuracy for different feature extraction methods

                                                                                                       Table (6): Comparison of recent models

5. Conclusion

As our reliance on interactions between humans and machines grows, recognizing human emotions has become crucial for advancing various industries. However, solely relying on facial expression analysis often proves inadequate for accurate emotion recognition. To improve detection accuracy, it is essential to use complementary methods that integrate multiple sources of information.

Automatic emotion classification remains challenging due to individual differences, even in controlled datasets like RAVDESS. Variations can arise among speakers of the same language and nationality. In this paper, we propose a novel method that combines facial expressions with two-channel speech signals to enhance emotion detection. Our framework employs an advanced deep learning-based classification model to extract features from both visual and verbal channels. It includes information fusion at the feature level and utilizes deep reinforcement learning to classify the combined auditory-visual emotional states. To further boost recognition performance, we incorporate the Grey Wolf Optimization (GWO) algorithm during the segmentation of extracted facial images. By mathematically modeling the hierarchical social behavior and hunting mechanisms of grey wolves, GWO optimizes the segmentation phase, enabling the identification of more discriminative regions within facial images. This optimized segmentation makes the features extracted from both speech and facial channels more representative, significantly enhancing the emotion recognition rate. Our evaluation using confusion matrices showed that the most common misclassification involved anger samples, which were frequently classified as fear or happiness. Despite these challenges, our proposed method achieved a remarkable accuracy of 89.44% on the RAVDESS dataset, surpassing previous approaches by leveraging GWO-based segmentation and advanced multi-modal fusion techniques. This demonstrates the effectiveness of our method in utilizing evolutionary algorithms to enhance deep learning models for dynamic emotion recognition tasks.

[1] Submission date:07, 02, 2024

Acceptance date:04, 10, 2025

Corresponding author: Sara Motamed, Department of Computer Engineering, Fouman and Shaft Branch, Islamic Azad University, Fouman, Iran

[i] https://datasets.activeloop.ai/docs/ml/datasets/ravdess-dataset/