A Bi-Directional GRU Architecture for the Self-Attention Mechanism: An Adaptable, Multi-Layered Approach with Blend of Word Embedding

Amit Pimpalkar; Jeberson Retna Raj

doi:10.46604/ijeti.2023.11510

Authors

Amit Pimpalkar School of Computing, Sathyabama Institute of Science and Technology, Chennai, India; Department of Computer Science and Engineering, Jhulelal Institute of Technology, Nagpur, India
Jeberson Retna Raj School of Computing, Sathyabama Institute of Science and Technology, Chennai, India

DOI:

https://doi.org/10.46604/ijeti.2023.11510

Keywords:

Bi-directional GRU, attention mechanism, deep learning, natural language processing, word embedding

Abstract

Sentiment analysis (SA) has become an essential component of natural language processing (NLP) with numerous practical applications to understanding “what other people think”. Various techniques have been developed to tackle SA using deep learning (DL); however, current research lacks comprehensive strategies incorporating multiple-word embeddings. This study proposes a self-attention mechanism that leverages DL and involves the contextual integration of word embedding with a time-dispersed bidirectional gated recurrent unit (Bi-GRU). This work employs word embedding approaches GloVe, word2vec, and fastText to achieve better predictive capabilities. By integrating these techniques, the study aims to improve the classifier’s capability to precisely analyze and categorize sentiments in textual data from the domain of movies. The investigation seeks to enhance the classifier’s performance in NLP tasks by addressing the challenges of underfitting and overfitting in DL. To evaluate the model’s effectiveness, an openly available IMDb dataset was utilized, achieving a remarkable testing accuracy of 99.70%.

References

X. Mao, S. Chang, J. Shi, F. Li, and R. Shi, “Sentiment-Aware Word Embedding for Emotion Classification,” Applied Sciences, vol. 9, no. 7, article no. 1334, April 2019.

D. Dangi, A. Bhagat, and D. K. Dixit, “Sentiment Analysis of Social Media Data Based on Chaotic Coyote Optimization Algorithm Based Time Weight-Adaboost Support Vector Machine Approach,” Concurrency and Computation: Practice and Experience, vol. 34, no. 3, article no. e6581, February 2022.

A. Pimpalkar and R. J. R. Raj, “Evaluation of Tweets for Content Analysis Using Machine Learning Models,” 12th International Conference on Computational Intelligence and Communication Networks, pp. 454-459, September 2020.

T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient Estimation of Word Representations in Vector Space,” https://arxiv.org/pdf/1301.3781.pdf, January 16, 2013.

W. Zhao, D. Fang, J. Zhang, Y. Zhao, X. Xu, X. Jiang, et al., “An Effective Framework for Semistructured Document Classification via Hierarchical Attention Model,” International Journal of Intelligent Systems, vol. 36, no. 9, pp. 5161‐5183, September 2021.

A. Pimpalkar and J. R. Raj R, “MBiLSTMGloVe: Embedding GloVe Knowledge into the Corpus Using Multi-Layer BiLSTM Deep Learning Model for Social Media Sentiment Analysis,” Expert Systems with Applications, vol. 203, article no. 117581, October 2022.

D. Dessí, M. Dragoni, G. Fenu, M. Marras, and D. R. Recupero, “Deep Learning Adaptation with Word Embeddings for Sentiment Analysis on Online Course Reviews,” Deep Learning-Based Approaches for Sentiment Analysis, Singapore: Springer, pp. 57-83, 2020.

L. Li, T. T. Goh, and D. Jin, “How Textual Quality of Online Reviews Affect Classification Performance: A Case of Deep Learning Sentiment Analysis,” Neural Computing and Applications, vol. 32, no. 9, pp. 4387-4415, May 2020.

N. D. Derra and D. Baier, “Working in Detail: How LSTM Hyperparameter Selection Influences Sentiment Analysis Results,” Archives of Data Science, Series A, vol. 6, no. 1, pp. 1-22, 2020.

R. Pushpakumar, K. S. Sakunthala Prabha, and P. N. Karthikayan, “For Movie Reviews, A Sentiment Analysis Using Long Short Term Memory Networks,” Turkish Journal of Computer and Mathematics Education, vol. 12, no. 9, pp. 1758-1766, April 2021.

P. Cen, K. Zhang, and D. Zheng, “Sentiment Analysis Using Deep Learning Approach,” Journal on Artificial Intelligence, vol. 2, no. 1, pp. 17-27, July 2020.

G. Liu and J. Guo, “Bidirectional LSTM with Attention Mechanism and Convolutional Layer for Text Classification,” Neurocomputing, vol. 337, pp. 325-338, April 2019.

Y. Ma, H. Fan, and C. Zhao, “Feature-Based Fusion Adversarial Recurrent Neural Networks for Text Sentiment Classification,” IEEE Access, vol. 7, pp. 132542-132551, 2019.

S. Li, R. Pan, H. Luo, X. Liu, and G. Zhao, “Adaptive Cross-Contextual Word Embedding for Word Polysemy with Unsupervised Topic Modeling,” Knowledge-Based Systems, vol. 218, article no. 106827, April 2021.

I. B. Ltaifa, L. Hlaoua, and L. B. Romdhane, “Hybrid Deep Neural Network-Based Text Representation Model to Improve Microblog Retrieval,” Cybernetics and Systems, vol. 51, no. 2, pp. 115-139, 2020.

B. Guo, C. Zhang, J. Liu, and X. Ma, “Improving Text Classification with Weighted Word Embeddings via a Multi-Channel TextCNN Model,” Neurocomputing, vol. 363, pp. 366-374, October 2019.

B. Jang, I. Kim, and J. W. Kim, “Word2vec Convolutional Neural Networks for Classification of News Articles and Tweets,” PLoS ONE, vol. 14, no. 8, article no. e0220976, 2019.

C. Wang, P. Nulty, and D. Lillis, “A Comparative Study on Word Embeddings in Deep Learning for Text Classification,” Proceedings of the 4th International Conference on Natural Language Processing and Information Retrieval, pp. 37-46, December 2020.

M. Kamyab, G. Liu, and M. Adjeisah, “Attention-Based CNN and Bi-LSTM Model Based on TF-IDF and GloVe Word Embedding for Sentiment Analysis,” Applied Sciences, vol. 11, no. 23, article no. 11255, December 2021.

M. Cho, J. Ha, C. Park, and S. Park, “Combinatorial Feature Embedding Based on CNN and LSTM for Biomedical Named Entity Recognition,” Journal of Biomedical Informatics, vol. 103, article no. 103381, March 2020.

P. Wu, X. Li, L. Chen, S. Ding, and S. Shen, “Sentiment Classification Using Attention Mechanism and Bidirectional Long Short-Term Memory Network,” Applied Soft Computing, vol. 112, article no. 107792, November 2021.

S. B. Akbar, K. Thanupillai, and S. Sundararaj, “Combining the Advantages of AlexNet Convolutional Deep Neural Network Optimized with Anopheles Search Algorithm based Feature Extraction and Random Forest Classifier for COVID-19 Classification,” Concurrency and Computation: Practice and Experience, vol. 34, no. 15, article no. e6958, July 2022.

A. T. Kabakus, “A Novel COVID-19 Sentiment Analysis in Turkish Based on the Combination of Convolutional Neural Network and Bidirectional Long-Short Term Memory on Twitter,” Concurrency and Computation: Practice and Experience, vol. 34, no. 22, article no. e6883, October 2022.

G. Kang, Y. Xiao, J. Liu, Y. Cao, B. Cao, X. Zhang, et al., “Tatt-BiLSTM: Web Service Classification with Topical Attention-Based BiLSTM,” Concurrency and Computation: Practice and Experience, vol. 33, no. 16, article no. e6287, August 2021.

J. Wang, Y. Zhang, L. C. Yu, and X. Zhang, “Contextual Sentiment Embeddings via Bi-Directional GRU Language Model,” Knowledge-Based Systems, vol. 235, article no. 107663, January 2022.

E. M. Dharma, F. L. Gaol, H. L. H. S. Warnars, and B. Soewito, “The Accuracy Comparison Among Word2vec, Glove, and Fasttext Towards Convolution Neural Network (CNN) Text Classification,” Journal of Theoretical and Applied Information Technology, vol. 100, no 2, pp. 349-359, January 2022.

M. Z. Naeem, F. Rustam, A. Mehmood, D. Mui Zzud, I. Ashraf, and G. S. Choi, “Classification of Movie Reviews Using Term Frequency-Inverse Document Frequency and Optimized Machine Learning Algorithms,” PeerJ Computer Science, vol. 8, article no. e914, 2022.

N. M. Alharbi, N. S. Alghamdi, E. H. Alkhammash, and J. F. Al Amri, “Evaluation of Sentiment Analysis via Word Embedding and RNN Variants for Amazon Online Reviews,” Mathematical Problems in Engineering, vol. 2021, article no. 5536560, September 2021.

M. Bilgin, “A New Statistics-Based Approach to Improve Word2Vec’s Sentiment Classification Success,” Selcuk University Journal of Engineering Sciences, vol. 20, no. 3, pp. 63-72, 2021.

R. Ranjan, D. Pandey, A. K. Rai, P. Singh, A. Vidyarthi, D. Gupta, et al., “A Manifold-Level Hybrid Deep Learning Approach for Sentiment Classification Using an Autoregressive Model,” Applied Sciences, vol. 13, no. 5, article no. 3091, March 2023.