A Review of Lip-Reading: From Datasets and Architectures to Cross-Linguistic Performance Gaps

Qian Hu; Kuryati Kipli; Tengku Mohd Afendi Zulcaffle; Yuan Liu; Jianrong Cao

doi:10.46604/aiti.2026.16377

Authors

Qian Hu Faculty of Engineering, University Malaysia Sarawak, Kota Samarahan, Sarawak, Malaysia/ Institute of Computer and Information Engineering, Qilu Institute of Technology, Shandong, China
Kuryati Kipli Faculty of Engineering, University Malaysia Sarawak, Kota Samarahan, Sarawak, Malaysia
Tengku Mohd Afendi Zulcaffle Faculty of Engineering, University Malaysia Sarawak, Kota Samarahan, Sarawak, Malaysia
Yuan Liu Institute of Computer and Information Engineering, Qilu Institute of Technology, Shandong, China
Jianrong Cao School of Information and Electrical Engineering, Shandong Jianzhu University, Shandong, China

DOI:

https://doi.org/10.46604/aiti.2026.16377

Keywords:

lip-reading, visual speech recognition, deep learning, cross-linguistic performance

Abstract

This paper aims to systematically review deep learning–driven lip-reading technologies, covering benchmark datasets, advances in model architecture, and cross-linguistic performance. Distinguished from existing surveys that predominantly enumerate methodologies in chronological sequence, this work consolidates the most extensive collection of 79 lip-reading datasets reported to date and establishes a problem-driven analytical framework to delineate the technological evolution from shallow feature extraction to global temporal modeling. Systematic analysis of the state of the art reveals a pronounced cross-linguistic performance disparity: state-of-the-art visual-only models attain 94.1% accuracy on standard English benchmarks, yet only 57.1% accuracy on Chinese benchmarks, representing a substantial performance gap of 37 percentage points rooted in intrinsic linguistic properties. This review identifies critical challenges and outlines promising research directions in lip-reading, therefore offering structured guidance for the continued development of multilingual visual speech recognition systems.

References

Z. Zhou, G. Zhao, X. Hong, and M. Pietikäinen, “A Review of Recent Advances in Visual Speech Decoding,” Image and Vision Computing, vol. 32, no. 9, pp. 590-605, 2014.

S. Mathulaprangsan, C.-Y. Wang, A. Z. Kusum, T.-C. Tai, and J.-C. Wang, “A Survey of Visual Lip Reading and Lip-Password Verification,” Proceedings of 2015 International Conference on Orange Technologies (ICOT), IEEE, pp. 22-25, 2015.

A. Fernandez-Lopez and F. M. Sukno, “Survey on Automatic Lip-Reading in the Era of Deep Learning,” Image and Vision Computing, vol. 78, pp. 53-72, 2018.

Y. Lu, J. Yan, and K. Gu, “Review on Automatic Lip Reading Techniques,” International Journal of Pattern Recognition and Artificial Intelligence, vol. 32, no. 7, article no. 1856007, 2018.

M. Hao, M. Mamut, N. Yadikar, A. Aysa, and K. Ubul, “A Survey of Research on Lipreading Technology,” IEEE Access, vol. 8, pp. 204518-204544, 2020.

L. Chen, G. Cui, Z. Kou, H. Zheng, and C. Xu, “What Comprises a Good Talking-Head Video Generation?: A Survey and Benchmark,” arXiv preprint arXiv:2005.03201, 2020.

S. Fenghour, D. Chen, K. Guo, B. Li, and P. Xiao, “Deep Learning-Based Automated Lip-Reading: A Survey,” IEEE Access, vol. 9, pp. 121184-121205, 2021.

C. Sheng, G. Kuang, L. Bai, C. Hou, Y. Guo, X. Xu, et al., “Deep Learning for Visual Speech Analysis: A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 9, pp. 6001-6022, 2024.

G. Pu and H. Wang, “Review on Research Progress of Machine Lip Reading,” The Visual Computer, vol. 39, no. 7, pp. 3041-3057, 2023.

N. Rathipriya and N. Maheswari, “A Comprehensive Review of Recent Advances in Deep Neural Networks for Lipreading with Sign Language Recognition,” IEEE Access, vol. 12, pp. 136846-136879, 2024.

S. Deshpande, K. Shirsath, A. Pashte, P. Loya, S. Shingade, and V. Sambhe, “A Comprehensive Survey of Advancement in Lip Reading Models: Techniques and Future Directions,” IET Image Processing, vol. 19, no. 1, article no. e70095, 2025.

J. R. Movellan, “Visual Speech Recognition with Stochastic Networks,” Advances in Neural Information Processing Systems, vol. 7, pp. 851-858, 1994.

O. Vanegas, K. Tokuda, and T. Kitamura, “Location Normalization of HMM-Based Lip-Reading: Experiments for the M2VTS Database,” Proceedings of 1999 International Conference on Image Processing (Cat. 99CH36348), IEEE, pp. 343-347, 1999.

K. Messer, J. Matas, J. Kittler, J. Luettin, and G. Maitre, “XM2VTSDB: The Extended M2VTS Database,” Second International Conference on Audio and Video-Based Biometric Person Authentication, Washington, D.C., USA, 1999.

Y. Xu, L. Du, G. Li, X. Zhang, and Z. Zhou, “Chinese Audiovisual Bimodal Speech Database CAVSR1. 0,” Acta Acoust. A Sinica, vol. 25, no. 1, pp. 42-49, 2000.

I. Matthews, T. F. Cootes, J. A. Bangham, S. Cox, and R. Harvey, “Extraction of Visual Features for Lipreading,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 2, pp. 198-213, 2002.

C. Sanderson, “The VidTIMIT Database,” IDIAP–Com 02-06, IDIAP, 2002.

E. K. Patterson, S. Gurbuz, Z. Tufekci, and J. N. Gowdy, “CUAVE: A New Audio-Visual Database for Multimodal Human-Computer Interface Research,” Proceedings of 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing, IEEE, pp. II-2017-II-2020, 2002.

E. Bailly-Bailliére, S. Bengio, F. Bimbot, M. Hamouz, J. Kittler, J. Mariéthoz, et al., “The BANCA Database and Evaluation Protocol,” Proceedings of International Conference on Audio-and Video-Based Biometric Person Authentication, Springer, pp. 625-638, 2003.

A. Ortega, F. Sukno, E. Lleida, A. F. Frangi, A. Miguel, L. Buera, et al., “AV@CAR: A Spanish Multichannel Multimodal Corpus for In-Vehicle Automatic Audio-Visual Speech Recognition,” Proceedings of International Conference on Language Resources and Evaluation (LREC), European Language Resources Association, pp. 763-766, 2004.

B. Lee, M. Hasegawa-Johnson, C. Goudeseune, S. Kamdar, J. Boley, S. Borys, et al., “AVICAR: Audio-Visual Speech Corpus in a Car Environment,” Proceedings of Interspeech Conference, ISCA, pp. 2489-2492, 2004.

J. Huang, G. Potamianos, J. Connell, and C. Neti, “Audio-Visual Speech Recognition Using an Infrared Headset,” Speech Communication, vol. 44, no. 1-4, pp. 83-96, 2004.

R. Goecke and J. B. Millar, “The Audio-Video Australian English Speech Data Corpus AVOZES,” Proceedings of the 8th International Conference on Spoken Language Processing, Interspeech, pp. 2525-2528, 2004.

T. J. Hazen, K. Saenko, C.-H. La, and J. R. Glass, “A Segment-Based Audio-Visual Speech Recognizer: Data Collection, Development, and Initial Experiments,” Proceedings of the 6th International Conference on Multimodal Interfaces, ACM Press, pp. 235-242, 2004.

N. A. Fox, B. A. O’Mullane, and R. B. Reilly, “VALID: A New Practical Audio-Visual Database, and Comparative Results,” Proceedings of International Conference on Audio- and Video-Based Biometric Person Authentication, Springer, pp. 777-786, 2005.

X Hong, H Yao, M Xu, “Bimodal Database and Its Material Segmentation for Lip-Reading Recognition on Sentence,” Computer Engineering and Applications, vol. 3, 2005.

P. Cisar, M. Železný, Z. Krňoul, J. Kanis, J. Zelinka, and L. Müller, “Design and Recording of Czech Speech Corpus for Audio-Visual Continuous Speech Recognition,” Proceedings of the Auditory-Visual Speech Processing International Conference 2005 (AVSP2005), pp. 93-96, 2005.

M. Cooke, J. Barker, S. Cunningham, and X. Shao, “An Audio-Visual Corpus for Speech Perception and Automatic Speech Recognition,” The Journal of the Acoustical Society of America, vol. 120, no. 5, pp. 2421-2424, 2006.

K. Kumar, T. Chen, and R. M. Stern, “Profile View Lip Reading,” Proceedings of 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP'07, IEEE, pp. IV-429-IV-432, 2007.

D. Petrovska-Delacrétaz, S. Lelandais, J. Colineau, L. Chen, B. Dorizzi, M. Ardabilian, et al., “The IV2 Multimodal Biometric Database (Including Iris, 2D, 3D, Stereoscopic, and Talking Face Data), and the IV2-2007 Evaluation Campaign,” Proceedings of 2008 IEEE Second International Conference on Biometrics: Theory, Applications and Systems, IEEE, pp. 1-7, 2008.

X. Lin, H. Yao, X. Hong, and Q. Wang, “HIT-AVDB-II: A New Multi-View and Extreme Feature Cases Contained Audio-Visual Database for Biometrics,” Proceedings of the 11th Joint International Conference on Information Sciences (JCIS 2008), Atlantis Press, pp. 357-363, 2008.

J. Trojanová, M. Hrúz, P. Campr, and M. Železný, “Design and Recording of Czech Audio-Visual Database with Impaired Conditions for Continuous Speech Recognition,” Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), European Language Resources Association (ELRA), pp. 1239-1243, 2008.

P. Lucey, G. Potamianos, and S. Sridharan, “Patch-Based Analysis of Visual Speech from Multiple Views,” Proceedings of the International Conference on Auditory-Visual Speech Processing 2008, AVISA, pp. 69-74, 2008.

S. J. Cox, R. W. Harvey, Y. Lan, J. L. Newman, and B.-J. Theobald, “The Challenge of Multispeaker Lip-Reading,” Proceedings of International Conference on Auditory-Visual Speech Processing (AVSP 2008), AVISA, pp. 179-184, 2008.

G. Zhao, M. Barnard, and M. Pietikäinen, “Lipreading with Local Spatiotemporal Descriptors,” IEEE Transactions on Multimedia, vol. 11, no. 7, pp. 1254-1265, 2009.

A. Vorwerk, X. Wang, D. Kolossa, S. Zeiler, and R. Orglmeister, “WAPUSK20—A Database for Robust Audiovisual Speech Recognition,” Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10), European Language Resources Association (ELRA), pp. 3016-3019, 2010.

A. Pass, J. Zhang, and D. Stewart, “An Investigation into Features for Multi-View Lipreading,” Proceedings of 2010 17th IEEE International Conference on Image Processing (ICIP), IEEE, pp. 2417-2420, 2010.

A. G. Chitu, K. Driel, and L. J. Rothkrantz, “Automatic Lip Reading in the Dutch Language Using Active Appearance Models on High Speed Recordings,” Proceedings of the 3rd International Conference on Text, Speech and Dialogue, Springer, pp. 259-266, 2010.

Y. Lan, B.-J. Theobald, R. W. Harvey, E.-J. Ong, and R. Bowden, “Improving Visual Features for Lip-Reading,” Proceedings of Auditory-Visual Speech Processing (AVSP), 2010.

S. Tamura, C. Miyajima, N. Kitaoka, T. Yamada, S. Tsuge, T. Takiguchi, et al., “CENSREC-1-AV: An Audio-Visual Corpus for Noisy Bimodal Speech Recognition,” Proceedings of Auditory-Visual Speech Processing (AVSP), ISCA, 2010.

Y. W. Wong, S. I. Ch’ng, K. P. Seng, L.-M. Ang, S. W. Chin, W. J. Chew, et al., “A New Multi-Purpose Audio-Visual UNMC-VIER Database with Multiple Variabilities,” Pattern Recognition Letters, vol. 32, no. 13, pp. 1503-1510, 2011.

Y. Benezeth, G. Bachman, G. Le-Jan, N. Souviraà-Labastie, and F. Bimbot, “BL-Database: A French Audiovisual Database for Speech Driven Lip Animation Systems,” INRIA, 2011.

V. Estellers and J.-P. Thiran, "Multipose Audio-Visual Speech Recognition," Proceedings of 2011 19th European Signal Processing Conference, IEEE, pp. 1065-1069, 2011.

M. Igras, B. Ziółko, and T. Jadczyk, “Audiovisual Database of Polish Speech Recordings,” Studia Informatica, vol. 33, no. 2B, pp. 163-172, 2012.

C. McCool, S. Marcel, A. Hadid, M. Pietikäinen, P. Matejka, J. Cernocký, et al., “Bi-modal Person Recognition on a Mobile Phone: Using Mobile Phone Data,” Proceedings of 2012 IEEE International Conference on Multimedia and Expo Workshops, IEEE, pp. 635-640, 2012.

S. Antar and A. Sagheer, “Audio-Visual Arabic Speech (AVAS) Database for Human-Computer Interaction Applications,” The International Journal of Advanced Research in Computer Science and Software Engineering, vol. 3, no. 9, 2013.

A. Rekik, A. Ben-Hamadou, and W. Mahdi, “A New Visual Speech Recognition Approach for RGB-D Cameras,” Proceedings of Image Analysis and Recognition: 11th International Conference, Springer, pp. 21-28, 2014.

N. Harte and E. Gillen, “TCD-TIMIT: An Audio-Visual Corpus of Continuous Speech,” IEEE Transactions on Multimedia, vol. 17, no. 5, pp. 603-615, 2015.

I. Anina, Z. Zhou, G. Zhao, and M. Pietikäinen, “Ouluvs2: A Multi-view Audiovisual Database for Non-rigid Mouth Motion Analysis,” Proceedings of 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), IEEE, pp. 1-5, 2015.

D. L. Howell, “Confusion Modelling for Lip-Reading,” Ph.D. dissertation, School of Computing Sciences, University of East Anglia, Norwich, UK, 2015.

Y. Mroueh, E. Marcheret, and V. Goel, “Deep Multimodal Learning for Audio-Visual Speech Recognition,” Proceedings of 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp. 2130-2134, 2015.

L. Schönherr, D. Orth, M. Heckmann, and D. Kolossa, “Environmentally Robust Audio-Visual Speaker Identification,” Proceedings of 2016 IEEE Spoken Language Technology Workshop (SLT), IEEE, pp. 312-318, 2016.

V. Verkhodanova, A. Ronzhin, I. Kipyatkova, D. Ivanko, A. Karpov, and M. Železný, “HAVRUS Corpus: High-Speed Recordings of Audio-Visual Russian Speech,” Proceedings of International Conference on Speech and Computer, Springer, pp. 338-345. 2016.

A. Czyzewski, B. Kostek, P. Bratoszewski, J. Kotus, and M. Szykulski, “An Audio-Visual Corpus for Multimodal Automatic Speech Recognition,” Journal of Intelligent Information Systems, vol. 49, no. 2, pp. 167-192, 2017.

A. Fernandez-Lopez, O. Martinez, and F. M. Sukno, “Towards Estimating the Upper Bound of Visual-Speech Recognition: The Visual Lip-Reading Feasibility Database,” Proceedings of 2017 12th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2017), IEEE, pp. 208-215, 2017.

S. Petridis, J. Shen, D. Cetin, and M. Pantic, “Visual-Only Recognition of Normal, Whispered and Silent Speech,” Proceedings of 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp. 6219-6223, 2018.

N. Alghamdi, S. Maddock, R. Marxer, J. Barker, and G. J. Brown, “A Corpus of Audio-Visual Lombard Speech with Frontal and Profile Views,” Journal of the Acoustical Society of America, vol. 143, no. 6, pp. EL523-EL529, 2018.

S. R. Livingstone and F. A. Russo, “The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A Dynamic, Multimodal Set of Facial and Vocal Expressions in North American English,” PLOS ONE, vol. 13, no. 5, article no. e0196391, 2018.

L. A. Elrefaei, T. Q. Alhassan, and S. S. Omar, “An Arabic Visual Dataset for Visual Speech Recognition,” Procedia Computer Science, vol. 163, pp. 400-409, 2019.

M. Gogate, K. Dashtipour, A. Adeel, and A. Hussain, “CochleaNet: A Robust Language-Independent Audio-Visual Model for Real-Time Speech Enhancement,” Information Fusion, vol. 63, pp. 273-285, 2020.

K. Wang, Q. Wu, L. Song, Z. Yang, W. Wu, C. Qian, et al., “Mead: A Large-Scale Audio-Visual Dataset for Emotional Talking-Face Generation,” Proceedings of European Conference on Computer Vision, Springer, pp. 700-717, 2020.

Y. Lu and H. Li, “Automatic Lip-Reading System Based on Deep Convolutional Neural Network and Attention-Based Long Short-Term Memory,” Applied Sciences, vol. 9, no. 8, article no. 1599, 2019.

M. Liu, L. Wang, K. A. Lee, H. Zhang, C. Zeng, and J. Dang, “Exploring Deep Learning for Joint Audio-Visual Lip Biometrics,” arXiv preprint arXiv:2104.08510, 2021.

M. Abdrakhmanova, A. Kuzdeuov, S. Jarju, Y. Khassanov, M. Lewis, and H. A. Varol, “Speakingfaces: A Large-Scale Multimodal Dataset of Voice Commands with Visual and Thermal Video Streams,” Sensors, vol. 21, no. 10, article no. 3465, 2021.

T. Exarchos, G. N. Dimitrakopoulos, A. G. Vrahatis, G. Chrysovitsiotis, Z. Zachou, and E. Kyrodimos, “Lip-Reading Advancements: A 3D Convolutional Neural Network/Long Short-Term Memory Fusion for Precise Word Recognition,” BioMedInformatics, vol. 4, no. 1, pp. 410-422, 2024.

M. T. R. Sahed, M. T. I. Aronno, H. Nyeem, M. A. Wahed, T. Ahsan, R. R. Islam, et al., “LipBengal: Pioneering Bengali Lip-Reading Dataset for Pronunciation Mapping through Lip Gestures,” Data in Brief, vol. 58, article no. 111254, 2025.

D. Estival, S. Cassidy, F. Cox, and D. Burnham, “AusTalk: An Audio-Visual Corpus of Australian English,” Proceedings of International Conference on Language Resources and Evaluation (LREC 2014), European Language Resources Association, pp. 3105-3109, 2014.

J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Lip Reading Sentences in the Wild,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6447-6456, 2017.

A. Nagrani, J. S. Chung, and A. Zisserman, “VoxCeleb: A Large-Scale Speaker Identification Dataset,” arXiv preprint arXiv:1706.08612, 2017.

J. S. Chung and A. Zisserman, “Lip Reading in Profile,” Proceedings of British Machine Vision Conference (BMVC 2017), BMVA Press, 2017.

T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Deep Audio-Visual Speech Recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 12, pp. 8717-8727, 2018.

T. Afouras, J. S. Chung, and A. Zisserman, “LRS3-TED: A Large-Scale Dataset for Visual Speech Recognition,” arXiv preprint arXiv:1809.00496, 2018.

A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, et al., “Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation,” arXiv preprint arXiv:1804.03619, 2018.

J. S. Chung, A. Nagrani, and A. Zisserman, “VoxCeleb2: Deep speaker recognition,” Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH 2018), pp. 1086-1090, 2018.

B. Shillingford, Y. Assael, M. W. Hoffman, T. Paine, C. Hughes, U. Prabhu, et al., “Large-Scale Visual Speech Recognition,” Proceedings of Interspeech 2019, pp. 4135-4139, 2019.

S. Yang, Y. Zhang, D. Feng, M. Yang, C. Wang, J. Xiao, et al., “LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild,” Proceedings of 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), IEEE, pp. 1-8, 2019.

Y. Zhao, R. Xu, and M. Song, “A Cascade Sequence-to-Sequence Model for Chinese Mandarin Lip Reading,” Proceedings of the 1st ACM International Conference on Multimedia in Asia, pp. 1-6, 2019.

T. Makino, H. Liao, Y. Assael, B. Shillingford, B. Garcia, O. Braga, et al., “Recurrent Neural Network Transducer for Audio-Visual Speech Recognition,” Proceedings of 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), IEEE, pp. 905-912, 2019.

J. Roth, S. Chaudhuri, O. Klejch, R. Marvin, A. Gallagher, L. Kaver, et al., “Ava Active Speaker: An Audio-Visual Dataset for Active Speaker Detection,” Proceedings of ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4492-4496. 2020.

K. R. Prajwal, R. Mukhopadhyay, V. P. Namboodiri, and C. V. Jawahar, “Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis,” Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13796-13805, 2020.

X. Chen, J. Du, and H. Zhang, “Lipreading with DenseNet and resBi-LSTM,” Signal, Image and Video Processing, vol. 14, no. 5, pp. 981-989, 2020.

Z. Zhang, L. Li, Y. Ding, and C. Fan, “Flow-Guided One-Shot Talking Face Generation with a High-Resolution Audio-Visual Dataset,” Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Press, pp. 3661-3670, 2021.

Y. Khassanov, S. Mussakhojayeva, A. Mirzakhmetov, A. Adiyev, M. Nurpeiissov, and H. Varol, “A Crowdsourced Open-Source Kazakh Speech Corpus and Initial Speech Recognition Baseline,” arXiv preprint arXiv:2009.10334, 2020.

E. Egorov, V. Kostyumov, M. Konyk, and S. Kolesnikov, “LRWR: Large-Scale Benchmark for Lip Reading in Russian Language,” arXiv preprint arXiv:2109.06692, 2021.

A. Lubitz, M. Valdenegro-Toro, and F. Kirchner, “The VVAD-LRS3 Dataset for Visual Voice Activity Detection,” arXiv preprint arXiv:2109.13789, 2021.

D. Ivanko, A. Axyonov, D. Ryumin, A. Kashevnik, and A. Karpov, “RUSAVIC Corpus: Russian Audio-Visual Speech in Cars,” Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC 2022), pp. 1555-1559, 2022.

A. Reece, G. Cooney, P. Bull, C. Chung, B. Dawson, C. Fitzpatrick, et al., “Advancing an Interdisciplinary Science of Conversation: Insights from a Large Multimodal Corpus of Human Speech,” arXiv preprint arXiv:2203.00674, 2022.

M. A. Haq, S.-J. Ruan, W.-J. Cai, and L. P.-H. Li, “Using Lip Reading Recognition to Predict Daily Mandarin Conversation,” IEEE Access, vol. 10, pp. 53481-53489, 2022.

C. Chen, D. Wang, and T. F. Zheng, “CN-CVS: A Mandarin Audio-Visual Dataset for Large Vocabulary Continuous Visual to Speech Synthesis,” Proceedings of ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp. 1-5, 2023.

M. Anwar, B. Shi, V. Goswami, W.-N. Hsu, J. Pino, and C. Wang, “Muavic: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation,” arXiv preprint arXiv:2303.00628, 2023.

J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, “Multimodal Deep Learning,” Proceedings of the 28th International Conference on Machine Learning (ICML 2011), Omnipress, pp. 689-696, 2011.

K. Noda, Y. Yamaguchi, K. Nakadai, H. G. Okuno, and T. Ogata, “Lipreading Using Convolutional Neural Network,” Interspeech, vol. 1, pp. 1149-1153, 2014.

M. Wand, J. Koutník, and J. Schmidhuber, “Lipreading with Long Short-Term Memory,” Proceedings of 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp. 6115-6119, 2016.

D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning Spatiotemporal Features with 3D Convolutional Networks,” Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), pp. 4489-4497, 2015.

A. Torfi, S. M. Iranmanesh, N. Nasrabadi, and J. Dawson, “3D Convolutional Neural Networks for Cross Audio-Visual Matching Recognition,” IEEE Access, vol. 5, pp. 22081-22091, 2017.

K. Xu, D. Li, N. Cassimatis, and X. Wang, “LCANet: End-to-End Lipreading with Cascaded Attention-CTC,” Proceedings of 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), IEEE, pp. 548-555, 2018.

K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.

T. Stafylakis and G. Tzimiropoulos, “Combining Residual Networks with LSTMs for Lipreading,” arXiv preprint arXiv:1703.04105, 2017.

X. Zhang, H. Gong, X. Dai, F. Yang, N. Liu, and M. Liu, “Understanding Pictograph with Facial Features: End-to-End Sentence-Level Lip Reading of Chinese,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 9211-9218, 2019.

D. Feng, S. Yang, S. Shan, and X. Chen, “Learn an Effective Lip Reading Model without Pains,” arXiv preprint arXiv:2011.07557, 2020.

Y. M. Assael, B. Shillingford, S. Whiteson, and N. de Freitas, “Lipnet: End-to-End Sentence-Level Lipreading,” arXiv preprint arXiv:1611.01599, 2016.

T. Arakane and T. Saitoh, “Efficient DNN Model for Word Lip-Reading,” Algorithms, vol. 16, no. 6, article no. 269, 2023.

C. Lea, R. Vidal, A. Reiter, and G. D. Hager, “Temporal Convolutional Networks: A Unified Approach to Action Segmentation,” Proceedings of European Conference on Computer Vision, Springer, pp. 47-54, 2016.

B. Martinez, P. Ma, S. Petridis, and M. Pantic, “Lipreading Using Temporal Convolutional Networks,” Proceedings of ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp. 6319-6323, 2020.

P. Ma, Y. Wang, J. Shen, S. Petridis, and M. Pantic, “Lip-Reading with Densely Connected Temporal Convolutional Networks,” Proceedings of the 2021 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), IEEE, pp. 2857-2866, 2021.

K. Prajwal, T. Afouras, and A. Zisserman, “Sub-word Level Lip Reading with Visual Attention,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5162-5172, 2022.

Z. Li, T. Lohrenz, M. Dunkelberg, and T. Fingscheidt, “Transformer-Based Lip-Reading with Regularized Dropout and Relaxed Attention,” Proceedings of 2022 IEEE Spoken Language Technology Workshop (SLT), IEEE, pp. 723-730, 2023.

Y.-H. Park, R.-H. Park, and H.-M. Park, “Swinlip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer,” Neurocomputing, vol. 639, article no. 130289, 2025.

L. Courtney and R. Sreenivas, “Learning from Videos with Deep Convolutional LSTM Networks,” arXiv preprint arXiv:1904.04817, 2019.

C. Wang, “Multi-Grained Spatio-Temporal Modeling for Lip-Reading,” arXiv preprint arXiv:1908.11618, 2019.

M. Luo, S. Yang, S. Shan, and X. Chen, “Pseudo-Convolutional Policy Gradient for Sequence-to-Sequence Lip-Reading,” Proceedings of 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), IEEE, pp. 273-280, 2020.

X. Zhao, S. Yang, S. Shan, and X. Chen, “Mutual Information Maximization for Effective Lip Reading,” Proceedings of 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), IEEE, pp. 420-427. 2020.

J. Xiao, S. Yang, Y. Zhang, S. Shan, and X. Chen, “Deformation Flow Based Two-Stream Network for Lip Reading,” Proceedings of 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), IEEE, pp. 364-370, 2020.

Y. Zhang, S. Yang, J. Xiao, S. Shan, and X. Chen, “Can We Read Speech Beyond the Lips? Rethinking Roi Selection for Deep Visual Speech Recognition,” Proceedings of 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), IEEE, pp. 356-363, 2020.

D. Ivanko, D. Ryumin, A. Kashevnik, A. Axyonov, and A. Karnov, “Visual Speech Recognition in a Driver Assistance System,” Proceedings of 2022 30th European Signal Processing Conference (EUSIPCO), IEEE, pp. 1131-1135, 2022.

A. Koumparoulis and G. Potamianos, “Accurate and Resource-Efficient Lipreading with Efficientnetv2 and Transformers,” Proceedings of ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp. 8467-8471, 2022.

P. Ma, Y. Wang, S. Petridis, J. Shen, and M. Pantic, “Training Strategies for Improved Lip-Reading,” Proceedings of ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp. 8472-8476, 2022.

H. Chen, W. Li, Z. Cheng, X. Liang, and Q. Zhang, “TCS-LipNet: Temporal & Channel & Spatial Attention-Based Lip Reading Network,” Proceedings of International Conference on Artificial Neural Networks, Springer Nature Switzerland, 2024.

H. Wang, B. Cui, Q. Yuan, G. Pu, X. Liu, and J. Zhu, “Mini-3DCvT: A Lightweight Lip-Reading Method Based on 3D Convolution Visual Transformer,” The Visual Computer, vol. 41, no. 3, pp. 1957-1969, 2025.

X. Wu, Z. Tan, Z. Cheng, and Y. Ru, “STDNet: Improved Lip Reading via Short-Term Temporal Dependency Modeling,” Virtual Reality & Intelligent Hardware, vol. 7, no. 2, pp. 173-187, 2025.

X. Zhang, J. Sun, P. Zhu, T. Xiao, M. Lao, and Y. Guo, “Mamba-Based Temporal Modeling for Event-Based Lip Reading,” Proceedings of 2025 6th International Conference on Computer Vision, Image and Deep Learning (CVIDL), IEEE, pp. 926-930, 2025.

M. Thomas, E. Fish, and R. Bowden, “VALLR: Visual ASR Language Model for Lip Reading,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, pp. 2846-2856, 2025.

R. Tapu, B. Mocanu, and I.-C. Chiva, “Multimodal Visual Speech Recognition for Under-Resource Languages via Cross-Modal Learning and Large Language Models,” Science and Technology (ROMJIST), vol. 29, no. 1, pp. 53-64, 2026.