Deaf and speech-impaired individuals rely on sign language — but sign language literacy among hearing people is rare. This gap forces a dependency on interpreters, limiting spontaneity, privacy, and access in everyday situations.
The client needed a technology layer that didn't just recognize signs, but rendered them with enough fidelity to be understood by someone with no sign language training — and did so in real time.
Rather than static image-based gesture displays, we chose 3D animated characters as the rendering layer. A 3D avatar communicates gesture, posture, and hand shape simultaneously — far closer to real sign language than flat diagrams or emoji-style shortcuts.
The system works bidirectionally: type text and watch it signed; record a gesture video and see it transcribed. Both flows run through a shared ML pipeline with a consistent latency target, so the experience feels like a conversation, not a translation tool.
A four-person team: full-stack, backend, AI/ML specialist, and frontend. The ML layer handles gesture recognition from camera input and sign-language sequence generation from text. The frontend renders the 3D avatar with smooth transitions between signs.
We built a real-time gesture chat feature on top — combining text, voice, and gestures in a single thread — with video call functionality in the pipeline for the next release cycle.
For the first time, deaf users can have a fluid conversation with someone who doesn't know a single sign. That's not a feature — that's independence.
Converts typed text into sequenced 3D sign language animations, rendered through a real-time avatar on device.
Camera-based ML pipeline that reads hand gestures and image inputs, transcribing them into readable text in real time.
A multimodal chat thread combining text, voice, and gestures — so conversations flow naturally across communication styles.
SanketBani is live on Android. The team is actively shipping — video call with real-time gesture recognition is the next milestone.
