Development and Evaluation of a 2D Convolutional Neural Network with Temporal Shift Module for Tagalog Visual Speech Recognition
Keywords:
Visual Speech Recognition, Lip Reading, Tagalog, 2D-CNN, Temporal Shift Module, Deep LearningAbstract
Visual Speech Recognition (VSR), or lip-reading, recognizes spoken content through visual information such as mouth and lip movements. This study will develop and evaluate a 2D Convolutional Neural Network (2D-CNN) combined with a Temporal Shift Module (TSM) for Tagalog visual speech recognition. The 2D-CNN will extract spatial features, while TSM will model temporal information across sequential video frames without using conventional 3D convolution. Connectionist Temporal Classification (CTC) will be used for sequence recognition and temporal alignment between visual features and corresponding Tagalog word sequences. The study will adopt an Explanatory Sequential Mixed-Methods Design with a predominantly quantitative approach involving dataset preparation, preprocessing, model development, training, validation, testing, and evaluation. Recognition performance will be assessed using Accuracy, Precision, Recall, F1-score, Matthews Correlation Coefficient (MCC), Character Error Rate (CER), and Word Error Rate (WER), while computational efficiency will be evaluated based on Training Time and Memory Requirements. A limited qualitative error analysis will then examine selected misclassified samples to identify observable mouth and lip movement patterns associated with recurring recognition errors. The study aims to provide empirical evidence on the recognition performance and computational efficiency of a 2D-CNN + TSM approach for Tagalog visual speech recognition.
References
Baloch, A., Ali, M., Hussain, L., Daud, A., & Dawood, H. (2026). Lip reading systems for Urdu alphabets in diverse environments. Scientific Reports, 16, Article 27134. https://doi.org/10.1038/s41598-026-53299-5
Bardhan, A., Singh, A., & Attri, S. H. (2024). CNN based real time detection of words from lip movements and automated into text. In K. Santosh, S. K. Sood, H. M. Pandey, & C. Virmani (Eds.), Advances in artificial-business analytics and quantum machine learning (Vol. 1191, pp. 87–102). Springer. https://doi.org/10.1007/978-981-97-2508-3_7
Barrios, A., & Garcia, R. (2023). Filipino Children’s Acquisition of Nominal and Verbal Markers in L1 and L2 Tagalog. Languages, 8(3), 188. https://doi.org/10.3390/languages8030188
Crespo, S. D. A., & Recario, R. N. C. (2024). Lip reading Filipino words using convolutional neural networks and recurrent neural networks. In K. Arai (Ed.), *Intelligent systems and applications: Proceedings of the 2023 Intelligent Systems Conference (IntelliSys) (Vol. 2, pp. 771–790). Springer. https://doi.org/10.1007/978-3-031-47724-9_51
Cruz, J. P. T., Abadiano, A. B., Sangilan, F. O., Requillo, E. G. T., & Juanatas, R. C. (2026). Utilizing residual network 50 convolutional neural network architecture for enhanced Philippine regional language classification on Jetson Orin Nano. Engineering Proceedings, 134(1), Article 2. https://doi.org/10.3390/engproc2026134002
Dweik, W., Altorman, S., & Ashour, S. (2022). Read my lips: Artificial intelligence word-level Arabic lipreading system. Egyptian Informatics Journal, 23(4), 1–12. https://doi.org/10.1016/j.eij.2022.06.001
Fu, Y., Lu, Y., & Ni, R. (2023). Chinese lip-reading research based on ShuffleNet and CBAM. Applied Sciences, 13(2), Article 1106. https://doi.org/10.3390/app13021106
Hao, B., Zhou, D., Li, X., Zhang, X., Xie, L., Wu, J., & Yin, E. (2025). LipGen: Viseme-guided lip video generation for enhancing visual speech recognition. In 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1–5). IEEE. https://doi.org/10.1109/ICASSP49660.2025.10889163
Haq, M. A., Ruan, S.-J., Cai, W.-J., & Li, L. P.-H. (2022). Using lip reading recognition to predict daily Mandarin conversation. IEEE Access, 10, 53481–53489. https://doi.org/10.1109/ACCESS.2022.3175867
Innocente, C., Boemio, M., Lorenzetti, G., Pulito, I., Romagnoli, D., Saponaro, V., Marullo, G., Ulrich, L., & Vezzetti, E. (2025). Deep learning-based lip-reading for vocal impaired patient rehabilitation. Computer Modeling in Engineering & Sciences, 143(2), 1355–1379. https://doi.org/10.32604/cmes.2025.063186
Li, H., Mamut, M., Yadikar, N., Zhu, Y., & Ubul, K. (2021). Channel enhanced temporal-shift module for efficient lipreading. In J. Feng, J. Zhang, M. Liu, & Y. Fang (Eds.), Biometric recognition: 15th Chinese Conference, CCBR 2021, Shanghai, China, September 10–12, 2021, proceedings (Vol. 12878, pp. 474–482). Springer. https://doi.org/10.1007/978-3-030-86608-2_52
Li, H., Yadikar, N., Zhu, Y., Mamut, M., & Ubul, K. (2022). Learning the relative dynamic features for word-level lipreading. Sensors, 22(10), Article 3732. https://doi.org/10.3390/s22103732
Li, Y., Hashim, A. S., Lin, Y., Nohuddin, P. N. E., Venkatachalam, K., & Ahmadian, A. (2024). AI-based visual speech recognition towards realistic avatars and lip-reading applications in the metaverse. Applied Soft Computing, 164, Article 111906. https://doi.org/10.1016/j.asoc.2024.111906
Lin, J., Gan, C., Wang, K., & Han, S. (2022). TSM: Temporal shift module for efficient and scalable video understanding on edge devices. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(5), 2760–2774. https://doi.org/10.1109/TPAMI.2020.3029799
Ma, P., Martinez, B., Petridis, S., & Pantic, M. (2021). Towards practical lipreading with distilled and efficient models. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 7608–7612). IEEE. https://doi.org/10.1109/ICASSP39728.2021.9415063
Ma, P., Petridis, S., & Pantic, M. (2022). Visual speech recognition for multiple languages in the wild. Nature Machine Intelligence, 4, 930–939. https://doi.org/10.1038/s42256-022-00550-z
Ma, Y., & Sun, X. (2025). Spatiotemporal feature enhancement for lip-reading: A survey. Applied Sciences, 15(8), Article 4142. https://doi.org/10.3390/app15084142
Oghbaie, M., Sabaghi, A., Hashemifard, K., & Akbari, M. (2021). Advances and challenges in deep lip reading. arXiv. https://doi.org/10.48550/arXiv.2110.07879
Panagos, I. I., Sfikas, G., & Nikou, C. (2024). Visual speech recognition using compact hypercomplex neural networks. Pattern Recognition Letters, 186, 1–7. https://doi.org/10.1016/j.patrec.2024.09.002
Pascual, A. D. V., Ayo, T. J. G., & Paglinawan, C. C. (2026). Tagalog lip-reading system using 3D convolutional neural network with bidirectional long short-term memory. Engineering Proceedings, 134(1), 55. https://doi.org/10.3390/engproc2026134055
Sheng, C., Kuang, G., Bai, L., Hou, C., Guo, Y., Xu, X., Pietikäinen, M., & Liu, L. (2024). Deep learning for visual speech analysis: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(9), 6001–6022. https://doi.org/10.1109/TPAMI.2024.3376710
Tamura, J., Itaya, Y., Hayashi, K., & Yamamoto, K. (2026). Statistical inference of the Matthews correlation coefficient for multiclass classification. Journal of Applied Statistics, 1–26. https://doi.org/10.1080/02664763.2026.2666305
Wang, H., Pu, G., & Chen, T. (2022). A lip reading method based on 3D convolutional vision transformer. IEEE Access, 10, 77205–77212. https://doi.org/10.1109/ACCESS.2022.3193231
Yeo, J. H., Kim, C. W., Kim, H., Rha, H., Han, S., Cheng, W.-H., & Ro, Y. M. (2025). Personalized lip reading: Adapting to your unique lip movements with vision and language. Proceedings of the AAAI Conference on Artificial Intelligence, 39(9), 9472–9480. https://doi.org/10.1609/aaai.v39i9.33026
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Andrea Q. Ysais, Mary Camille S. Santos, Paolo Emerson L. Cordero, Phil Andrei B. Sison, Mikayla Darlene B. Domingo (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
