Building Sign Language Recognition with Deep Learning

Indian Sign Language (ISL) recognition is a challenging computer vision problem. In this post, I describe the system I built at PICT Pune that was published at IEEE ICONAT 2024.

The Challenge

ISL has over 3,000 signs, many of which are dynamic and require motion context to understand. Most existing systems only handled static signs.

Our Approach

• ▸**Detection**: YOLO-NAS for real-time hand and body detection
• ▸**Keypoints**: MediaPipe for skeletal keypoint extraction (21 hand landmarks + 33 body pose points)
• ▸**Classification**: LSTM for temporal sequence modeling of dynamic signs
• ▸**Translation**: NLP pipeline to convert recognized signs to natural language

Achievements

• ▸15% improvement in recognition accuracy over baseline
• ▸75ms reduction in end-to-end translation latency
• ▸Multi-person support (up to 3 simultaneous signers)