Deep Learning Based Survey For Eye Impaired Patients With Sign Language Analysis
DOI:
https://doi.org/10.52783/jns.v14.2917Abstract
Numerous people with impairments, such as the blind, deaf, as well as dumb, are seen by us daily. several interaction methods available for both the hard-of-hearing and public is sign language. However, the sign phrases and movements used by the deaf and dumb are difficult for normal individuals to grasp. The sign language that people with disabilities produce can be translated in an expression which is understandable by others using a variety of techniques. The research focuses on different approaches for picture capture, initial processing, segmenting movements of the hands, obtaining features, and categorization. The purpose of this work is to investigate and analyse the methodologies utilized in SLR networks, as well as the methods of classification applied, and then suggest the approach with the greatest promise for further study. A couple of the recently offered efforts, along with combined approaches including deep learning, notably improve approaches for classification because of the most recent developments in categorization approaches. The focus of this work revolves around identifying techniques used in previous Sign Language Identification research. This study indicates that earlier investigations, which incorporate adaptations, examined HMM-based approaches extensively. During the last five years, deep learning using neural networks based on convolution gained popularity.
Downloads
Metrics
References
G.A. Rao, P.V.V. Kishore, Selfie sign language recognition with multiple features on adaboost multilabel multiclass classifier, J. Eng. Sci. Technol. 13 (8) (2018) 2352–2368.
P.V.V. Kishore, P.R. Kumar, A video based Indian Sign Language Recognition System (INSLR) using wavelet transform and fuzzy logic, Int. J. Eng. Technol. 4 (5) (2012) 537.
https://gesture.chalearn.org/2014-looking-at-people-challenge.
Sylvie C.W. Ong, Surendra Ranganath, Automatic sign language analysis: a survey and the future beyond lexical meaning, IEEE Trans. Pattern Anal. Mach. Intell. 27 (2005) 6, https://doi.org/10.1109/TPAMI.2005.112 (June 2005), 873–891.
M. Eslami, M. Karami, S. Tabarestani, F. Torkamani-Azar, S. Eslami, C. Meinel, SignCol: open-source software for collecting sign language gestures, in: 2018 IEEE 9th International Conference on Software Engineering and Service Science (ICSESS), Beijing, China, 2018, pp. 365–369.
Dong Cao, M.C. Leu, Z. Yin, American sign language alphabet recognition using Microsoft Kinect, in: 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW, Boston, MA, 2015, pp. 44–52.
P.V.V. Kishore, M.V.D. Prasad, D.A. Kumar, A.S.C.S. Sastry, Optical flow hand tracking and active contour hand shape features for continuous sign language recognition with artificial neural networks, in: 2016 IEEE 6th International Conference on Advanced Computing (IACC), Bhimavaram, 2016, pp. 346–351.
B. Shi, et al., American sign language fingerspelling recognition in the wild, in: 2018 IEEE Spoken Language Technology Workshop (SLT), Athens, Greece, 2018, pp. 145–152.
M. Krishnaveni, V. Radha, Classifier fusion based on Bayes aggregation method for Indian sign language datasets, Procedia Eng. 30 (2012) 1110–1118.
S.S. Shivashankara, S. Srinath, American sign language recognition system: an optimal approach, Int. J. Image Graph. Signal Process. (2018).
Kshitij Bantupalli, Ying Xie, American sign language recognition using machine learning and computer vision, Master of Science in Computer Science Theses 21 (2019).
Nandy, J.S. Prasad, S. Mondal, P. Chakraborty, G.C. Nandi, Recognition of isolated indian sign language gesture in real time, Inf. Process. Manag. (2010) 102–107.
Yang Su, Qing Zhu, Continuous Chinese sign language recognition with CNN- LSTM, in: Proc. SPIE 10420, Ninth International Conference on Digital Image Processing (ICDIP 2017), 21 July 2017, p. 104200F, https://doi.org/10.1117/12.2281671.
Q. Xiao, Y. Zhao, W. Huan, Multi-sensor data fusion for sign language recognition based on dynamic Bayesian network and convolutional neural network, Multimed. Tool. Appl. 78 (2019) 15335–15352, https://doi.org/10.1007/s11042-018-6939-8.
N.C. Camgoz, S. Hadfield, O. Koller, R. Bowden, SubUNets: end-to-end hand shape and continuous sign language recognition, in: 2017 IEEE International Conference on Computer Vision (ICCV), Venice, 2017, pp. 3075–3084.
Kika, A. Koni, Hand gesture recognition using convolutional neural network and histogram of oriented gradients features, in: CEUR Workshop Proceedings, vol. 2280, CEUR-WS, 2018, pp. 75–79.
P.C. Badhe, V. Kulkarni, Indian sign language translator using gesture recognition algorithm, in: 2015 IEEE International Conference on Computer Graphics, Vision and Information Security (CGVIS), Bhubaneswar, 2015, pp. 195–200.
P.A. Nanivadekar, V. Kulkarni, Indian sign language recognition: database creation, hand tracking and segmentation, in: 2014 International Conference on Circuits, Systems, Communication and Information Technology Applications, CSCITA, Mumbai, 2014, pp. 358–363.
H. Lilha, D. Shivmurthy, Analysis of pixel level features in recognition of real life dual-handed sign language data set, in: Recent Trends in Information Systems (ReTIS), 2011 International Conference on, IEEE, 2011, December, pp. 246–251.
J. Nagi, et al., Max-pooling convolutional neural networks for vision-based hand gesture recognition, in: 2011 IEEE International Conference on Signal and Image Processing Applications (ICSIPA), Kuala Lumpur, 2011, pp. 342–347, https://doi. org/10.1109/ICSIPA.2011.6144164.
K.M. Lim, A.W.C. Tan, S. Tan, A feature covariance matrix with serial particle filter for isolated sign language recognition, Expert Syst. Appl. 54 (2016) 208–218, https://doi.org/10.1016/j.eswa.2016.01.047.
M. Mohandes, M. Deriche, U. Johar, S. Ilyas, A signer-independent Arabic sign language recognition system using face detection, geometric features, and a hidden Markov model, Comput. Electr. Eng. 38 (2) (2012) 422–433, https://doi.org/ 10.1016/j.compeleceng.2011.10.013.
B.M. Chethana Kumara, H.S. Nagendraswamy, R Lekha Chinmayi, Spatial relationship based features for Indian sign language recognition, International Journal of Computing, communications & Instrumentation Engineering 3 (2) (2016), 2349- 1469.
S.G.M. Almeida, F.G. Guimar˜aes, J.A. Ramírez, Feature extraction in brazilian sign language recognition based on phonological structure and using RGB-d sensors, Expert Syst. Appl. 41 (16) (2014) 7259–7271, https://doi.org/10.1016/j. eswa.2014.
Eriglen Gani, Alda Kika, Albanian sign language (AlbSL) number recognition from both hand’s gestures acquired by Kinect sensors, Int. J. Adv. Comput. Sci. Appl. 7 (2016) 7, 2016.
M. Boulares, M. Jemni, 3D motion trajectory analysis approach to improve sign language 3d-based content recognition, Procedia Comput. Sci. 13 (2012) 133–143.
T. Raghuveera, R. Deepthi, R. Mangalashri, et al., A depth-based Indian sign language recognition using Microsoft Kinect, Sa¯dhana¯ 45 (2020) 34, https://doi. org/10.1007/s12046-019-1250-6.
Rúbia Reis Guerra, Rezende, Tamires Martins Guimar˜aes, Frederico Gadelha, Sílvia Grasiella Moreira Almeida, Facial expression analysis in Brazilian sign language for sign recognition, in: NATIONAL MEETING OF ARTIFICIAL AND COMPUTATIONAL INTELLIGENCE, ENIAC, 2018.
Becky Sue Parton, sign language recognition and translation: a multidisciplined approach from the field of artificial intelligence, J. Deaf Stud. Deaf Educ. 11 (1) (2006) 94–101, https://doi.org/10.1093/deafed/enj003. Winter.
http://www.massey.ac.nz/~albarcza/gesture_dataset2012.html.
http://vlm1.uta.edu/~srujana/ASLID/ASL_Image_Dataset.html.
https://image-net.org/challenges/LSVRC/2010/.
J. Forster, C. Schmidt, O. Koller, M. Bellgardt, H. Ney, Extensions of the sign language recognition and translation Corpus RWTH-PHOENIX-weather, in: International Conference on Language Resources and Evaluation, LREC, 2014.
https://www-i6.informatik.rwth-aachen.de/~koller/1miohands-data/.
A.A. Ahmed, S. Aly, Appearance-based Arabic sign language recognition using hidden Markov models, in: 2014 International Conference on Engineering and Technology, ICET, Cairo, 2014, pp. 1–6.
https://www.phonetik.uni-muenchen.de/forschung/Bas/SIGNUM/.
C. Savur, F. Sahin, American Sign Language Recognition system by using surface EMG signal, in: 2016 IEEE International Conference on Systems, Man, and Cybernetics, SMC, Budapest, 2016, 002872-002877.
D. Soydaner, A comparison of optimization algorithms for deep learning, Int. J. Pattern Recogn. Artif. Intell. 34 (13) (2020), 2052013.
R. Akmeliawati, M.P. Ooi, Y.C. Kuang, Real-time Malaysian sign language translation using colour segmentation and neural network, in: 2007 IEEE Instrumentation & Measurement Technology Conference IMTC 2007, Warsaw, 2007, pp. 1–6.
P.K. Athira, C.J. Sruthi, A. Lijiya, A signer independent sign language recognition with co-articulation elimination from live videos: an indian scenario, J. King Saud Univ. Comput. Inf. Sci. (2019), https://doi.org/10.1016/j.jksuci.2019.05.002.
G.A. Rao, K. Syamala, P.V.V. Kishore, A.S.C.S. Sastry, Deep convolutional neural networks for sign language recognition, in: 2018 Conference on Signal Processing and Communication Engineering Systems (SPACES), Vijayawada, 2018, pp. 194–197, https://doi.org/10.1109/SPACES.2018.8316344.
Yongsen Ma, Gang Zhou, Shuangquan Wang, Hongyang Zhao, Woosub Jung, SignFi: sign language recognition using WiFi, Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 2 (1) (2018) 21. Article 23 (Mar. 2018).
D. Rathi, Optimization of Transfer Learning for Sign Language Recognition Targeting Mobile Platform, 2018 arXiv preprint arXiv:1805.06618.
S. Yang, Q. Zhu, Video-based Chinese sign language recognition using convolutional neural network, in: 2017 IEEE 9th International Conference on Communication Software and Networks, ICCSN, Guangzhou, 2017, pp. 929–934, https://doi.org/10.1109/ICCSN.2017.8230247.
V. Bheda, D. Radpour, Using Deep Convolutional Networks for Gesture Recognition in American Sign Language, 2017 arXiv preprint arXiv:1710.06836.
P.V.V. Kishore, K.B.N.S.K. Chaitanya, G.S.S. Shravani, Teja Maddala, Kiran Eepuri, D. Anil Kumar, DSLR-net a Depth Based Sign Language Recognition Using Two Stream Convents, vol. 8, 2019, pp. 765–773.
Jie Huang, Wengang Zhou, Houqiang Li, Weiping Li, Sign Language Recognition using 3D convolutional neural networks, in: 2015 IEEE International Conference on Multimedia and Expo, ICME, Turin, 2015, pp. 1–6.
Arif-Ul-Islam, S. Akhter, Orientation hashcode and articial neural network based combined approach to recognize sign language, in: 2018 21st International Conference of Computer and Information Technology, ICCIT, Dhaka, Bangladesh, 2018, pp. 1–5.
W. Tao, M.C. Leu, Z. Yin, American sign language alphabet recognition using convolutional neural networks with multiview augmentation and inference fusion, Eng. Appl. Artif. Intell. 76 (2018) 202–213.
Zhi-jie Liang, Sheng-bin Liao, Bing-zhang Hu, 3D convolutional neural networks for dynamic sign language recognition, Comput. J. 61 (11) (November 2018) 1724–1736, https://doi.org/10.1093/comjnl/bxy049.
Nikhil Kasukurthi, Brij Rokad, Shiv Bidani, Aju Dennisan American Sign Language Alphabet Recognition Using Deep Learning, 2014.
Biyi Fang, Jillian Co, Mi Zhang, DeepASL: enabling ubiquitous and non-intrusive word and sentence-level sign language translation, in: Proceedings of the 15th ACM Conference on Embedded Networked Sensor Systems (SenSys ’17), 2017.
D. Avola, M. Bernardi, L. Cinque, G.L. Foresti, C. Massaroni, Exploiting recurrent neural networks and Leap motion controller for the recognition of sign language and semaphoric hand gestures, IEEE Trans. Multimed. 21 (1) (Jan. 2019) 234–245.
Wadhawan, P. Kumar, Deep Learning-Based Sign Language Recognition System for Static Signs, Neural Comput & Applic, 2020, https://doi.org/10.1007/s00521- 019-04691-y.
N.C. Camgoz, S. Hadfield, O. Koller, H. Ney, R. Bowden, Neural sign language translation, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018, pp. 7784–7793.
Srujana Gattupalli, Amir Ghaderi, Vassilis Athitsos, Evaluation of deep learning-based pose estimation for sign language recognition, in: Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments (PETRA ’16), Association for Computing Machinery, New York, NY, USA, 2016, https://doi.org/10.1145/2910674.2910716. Article 12, 1–7.
Bowen Shi, Aurora Martinez Del Rio, Jonathan Keane, Diane Brentari, Greg Shakhnarovich, Karen Livescu, Fingerspelling Recognition in the Wild with Iterative Visual Attention, 2019.
O. Koller, S. Zargaran, H. Ney, et al., Deep sign: enabling Robust statistical continuous sign language recognition via hybrid CNN-HMMs, Int. J. Comput. Vis. 126 (2018) 1311–1325, https://doi.org/10.1007/s11263-018-1121-3.
R. Cui, H. Liu, C. Zhang, Recurrent convolutional neural networks for continuous sign language recognition by staged optimization, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Honolulu, HI, 2017, pp. 1610–1618.
O. Koller, S. Zargaran, H. Ney, Re-sign: Re-aligned end-to-end sequence modelling with deep recurrent CNN-HMMs, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Honolulu, HI, 2017, pp. 3416–3424.
Brandon Garcia, Sigberto Viesca, Real-time American sign language recognition with convolutional neural networks, in: Convolutional Neural Networks for Visual Recognition at Stanford University, 2016.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Reed Scott, Dragomir Anguelov, Dumitru Erhan, Vanhoucke Vincent, Andrew Rabinovich, Going deeper with convolutions, in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015, pp. 1–9. Boston, Ma, USA.
K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, in: International Conference on Learning Representations, ICLR, 2015.
S.J. Pan, Q. Yang, et al., A Survey on Transfer Learning, IEEE Transactions on knowledge and data engineering, 2010.
Junfu Pu, Wengang Zhou, Houqiang Li, Dilated convolutional network with iterative optimization for coutinuous sign language recognition, in: International Joint Conference on Artificial Intelligence, IJCAI, 2018, pp. 885–891.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Weyand Tobias, Marco Andreetto, Hartwig Adam, MobileNets: efficient convolutional neural networks for mobile vision applications. https://arxiv. org/abs/1704.04861, 2017.
B. Kang, S. Tripathi, T. Nguyen, "Real-time sign language fingerspelling recognition using convolutional neural networks from depth map", Pattern Recognition 2015 3rd IAPR Asian Conference on, Nov. 2015.
P. Molchanov, S. Gupta, K. Kim, J. Kautz, Hand gesture recognition with 3D convolutional neural networks, in: 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW, Boston, MA, 2015, pp. 1–7, https://doi. org/10.1109/CVPRW.2015.7301342.
D. Guo, W. Zhou, H. Li, M. Wang, Hierarchical LSTM for sign language translation, in: AAAI Conference on Artificial Intelligence, North America, apr. 2018.
S. Masood, H.C. Thuwal, A. Srivastava, S. Satapathy, V. Bhateja, S. Das, American sign language character recognition using convolution neural network, in: Smart Computing and Informatics. Smart Innovation Systems and Technologies, vol. 78, Springer, Singapore, 2018.
T. Liu, W. Zhou, H. Li, Sign Language Recognition with long short-term memory, in: 2016 IEEE International Conference on Image Processing, ICIP, Phoenix, AZ, 2016, pp. 2871–2875.
Y. Liao, P. Xiong, W. Min, W. Min, J. Lu, Dynamic Sign Language Recognition based on video sequence with BLSTM-3D residual networks, IEEE Access 7 (2019) 38044–38054.
Alexander Toshev, Christian Szegedy, Deeppose: human pose estimation via deep neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014.
sign language MNIST, Kaggle. https://www.kaggle.com/datamunge/sign-langua ge-mnist/, 2017.
S. Yang, Q. Zhu, Video-based Chinese Sign Language Recognition using convolutional neural network, in: IEEE 9th International Conference on Communication Software and Networks (ICCSN), Guangzhou, 2017, pp. 929–934.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
You are free to:
- Share — copy and redistribute the material in any medium or format
- Adapt — remix, transform, and build upon the material for any purpose, even commercially.
Terms:
- Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
