Sign Language to Text and Speech Conversion
Keywords:
hearing impairments, American Sign Language(ASL), Computer Vision, Real-time System, Convolutional Neural Networks (CNN)Abstract
Sign language is a rich and deeply ingrained form of communication that has been used for centuries to bridge communication gaps between individuals with hearing impairments and the hearing world. Its historical significance and the innate human need for expression make it a fascinating subject of study. In the modern age, technology has evolved up new possibilities for enhancing sign language communication through innovative methods. We have embarked on a journey to harness the power of neural networks to develop a real-time system for finger spelling in American Sign Language (ASL). This endeavour is driven by the recognition that ASL is not only one of the oldest but also one of the commonly used natural forms of language expression. By leveraging the capabilities of convolutional neural networks (CNNs), we aim to revolutionize the way we perceive and interpret ASL gestures. Our approach involves automatic gesture recognition from camera images, a field brimming with potential in the realm of computer vision. Using a CNN-based methodology, we seek to decode the intricate hand gestures that are intrinsic to human communication. Central to our methodology is the extraction of critical information, such as hand position and orientation, from camera-captured images. The Profound Impact of Sign Language and the Role of Technology in Enhancing Communication Sign language stands as one of the most expressive and meaningful forms of human communication. As a visually-driven language developed over centuries, it serves as a vital bridge for individuals who are deaf or hard of hearing, enabling them to connect, share ideas, and express emotions in deeply nuanced ways. Far from being a simple system of hand movements, sign language reflects a rich cultural and linguistic heritage.
Downloads
References
A. Adeyanju, O. O. Bello, and M. A. Adegboye conducted a comprehensive review and analysis of machine learning techniques applied to sign language recognition, which was published in Intelligent Systems and Applications, volume 12, in November 2021, under article number 200056. The DOI for this work is 10.1016/j.iswa.2021.200056.
. Auephanwiriyakul, S. Phitakwinai, W. Suttapak, P. Chanda, and N. Theera-Umpon presented a method for translating Thai sign language utilizing Scale Invariant Feature Transform and Hidden Markov Models in Pattern Recognition Letters, volume 34, issue 11, pages 1291–1298, in August 2023.
E.-S.-M. El-Alfy and H. Luqman provided a thorough survey and taxonomy of sign language research in Engineering Applications of Artificial Intelligence, volume 114, in September 2022, under article number 105198. The DOI is 10.1016/j.engappai.2022.105198.
M. Al-Qurishi, T. Khalid, and R. Souissi discussed current techniques, benchmarks, and unresolved issues related to deep learning in sign language recognition in IEEE Access, volume 9, pages 126917–126951, in 2021. The DOI for this publication is 10.1109/ACCESS.2021.3110912.
B. Bauer and H. Hienz highlighted important features for video-based continuous sign language recognition at the 4th IEEE International Conference on Automatic Face and Gesture Recognition, held in March 2020, with their findings on pages 440–445.
S. Bai, J. Zico Kolter, and V. Koltun conducted an empirical evaluation comparing generic convolutional and recurrent networks for sequence modeling, which was made available in 2018 under arXiv:1803.01271.
M. J. Cheok, Z. Omar, and M. H. Jaward reviewed various techniques for recognizing hand gestures and sign language in the International Journal of Machine Learning and Cybernetics, volume 10, issue 1, pages 131–153, in January 2019.
N. Cihan Camgöz, O. Koller, S. Hadfield, and R. Bowden introduced "Sign Language Transformers," which provide a unified approach to end-to-end sign language recognition and translation at the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) in June 2020, appearing on pages 10020–10030.
M. Dilsizian, P. Yanovich, S. Wang, C. Neidle, and D. Metaxas proposed a novel framework for sign language recognition that incorporates 3D handshape identification and linguistic modeling, presented at the 9th International Conference on Language Resources and Evaluation in 2018.
Q. Fu, J. Fu, J. Guo, S. Guo, and X. Li discussed gesture recognition leveraging a BP neural network and data glove at the IEEE International Conference on Mechatronics and Automation (ICMA) in October 2020.
G. Fang and W. Gao presented a system based on SRN/HMM designed for signer-independent continuous sign language recognition at the 5th IEEE International Conference on Automatic Face and Gesture Recognition in November 2022, on pages 312–317.
G. Fang, W. Gao, and D. Zhao explored large-vocabulary continuous sign language recognition using transition-movement models in IEEE Transactions on Systems, Man, and Cybernetics: Systems and Humans, volume 37, issue 1, pages 1–9, in January 2020..
W. Gao, G. Fang, D. Zhao, and Y. Chen developed a recognition system for Chinese sign language based on SOFM/SRN/HMM, published in Pattern Recognition, volume 37, issue 12, pages 2389–2402, in December 2018.
]K. He, X. Zhang, S. Ren, and J. Sun introduced deep residual learning for image recognition at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in June 2019, with their work appearing on pages 770–778.
Seetharaman, K., and N. Palanivel. 2013. “Texture Characterization, Representation, Description, and Classification Based on Full Range Gaussian Markov Random Field Model with Bayesian Approach.” International Journal of Image and Data Fusion 4 (4): 342–62. doi:10.1080/19479832.2013.804007.
S.-H. Yu, C.-L. Huang, S.-C. Hsu, H.-W. Lin, and H.-W. Wang presented a vision-based continuous sign language recognition approach using product HMM at the 1st Asian Conference on Pattern Recognition in November 2021, on pages 510–514.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
You are free to:
- Share — copy and redistribute the material in any medium or format
- Adapt — remix, transform, and build upon the material for any purpose, even commercially.
Terms:
- Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
