Hackathon project / Accessibility prototype, Indian Sign Language
Avya
A service-counter communication prototype that turns sign language into spoken words, so a person who cannot speak can be heard by the official in front of them.
- Status
- Prototype. Recognises 5 signs. Built for the Sankalp Setu hackathon
- Year
- 2026
Spoken aloud
A representative design of the idea, not a screenshot of a finished product.
My role
- Team lead for Team RaShVa, a five-member team.
- Product concept, interface direction and prototype development with the team.
At public service counters, people who are hearing impaired or verbally challenged often have no easy way to be understood.
Why it mattered
Being understood at a counter decides whether someone gets the service they came for.
- Camera and screen face the user. A second screen faces the official and speaks the message aloud.
- Start with Hindi as the spoken language, then extend to regional languages.
What it does
- Hand tracking turns signs into recognised words.
- The current prototype is trained on five signs and runs on a single laptop.
- Planned for later phases: wider sign vocabulary, regional languages and sign animation for spoken replies.
What I wanted to build: A way for a person who cannot speak to be understood at a service counter.
How it came together
- Framed the two-screen counter setup with the team.
- Used prompts and iteration to wire hand tracking to recognised signs and spoken output.
- Trained and tested on five signs, and kept the claim to that.
What I did
- Led the team and the product direction.
- Made the design and product decisions, and reviewed what the tools produced.
- Tested, debugged and decided what to keep, rewrite or cut.
- Configured the services and handled deployment.
Where AI tools helped
- AI tools drafted and refactored code from structured prompts, and I iterated on the results.
Services connected
- MediaPipe hand tracking, web app, Supabase and Vercel.
Working today
- Recognition of five signs and spoken output on a single laptop.
Still prototype or planned
- The wider sign vocabulary, regional languages and sign animation are planned, not built.
Written from my own account of the build.
- A small trained vocabulary limits what the prototype can understand today.
- Learning enough Indian Sign Language to design responsibly.
A working proof of concept built in the context of the Sankalp Setu Student AI Hackathon. It is not a deployed hospital or government solution, and it does not recognise the full language.
Accessibility work needs the people it serves in the room, not just a clever model.
Exploring an accessibility or computer vision idea?