GestureKey started as a straightforward research question: can I build a computer vision system that reliably classifies sign language gestures from image data, and eventually get it to a point where it's useful to someone who is deaf or hard of hearing? The answer, I've found, is yes — but the path there is messier than any paper I read beforehand suggested.
The assumption I got wrong
Before I wrote a single line of preprocessing code, I spent a lot of time reading about model architectures. CNNs, transfer learning from ImageNet, attention mechanisms. I had a notebook full of comparisons. I thought the key decision was going to be which model to use. It wasn't.
The key decision was how to handle the data. Specifically, how to deal with the fact that publicly available sign language image datasets are inconsistent in ways that aren't obvious until your model starts failing at inference time on images that look completely normal to a human eye.
What inconsistent data actually looks like
When I say inconsistent, I mean several things:
- Lighting variation — some images in the same dataset were taken under bright studio light, others in natural daylight, others under artificial warm lighting. A model trained on this mix learns spurious correlations between brightness and class labels without you realising it.
- Hand angle and perspective — the same gesture photographed from slightly different angles can look very different at the pixel level. If your training set over-represents one angle for a particular gesture, your model learns that angle as a feature, not the gesture itself.
- Background noise — some images had clean white backgrounds, others had real-world clutter. The model picks up on background statistics if you're not careful, especially in smaller datasets where there aren't enough samples to average out irrelevant features.
None of this is surprising in retrospect. But when you're running your first cross-validation loop and seeing 94% accuracy, it's easy to feel like the hard work is done. It isn't. That 94% is often telling you how well your model has memorised the training distribution — not how well it generalises to a real hand in real lighting held by a real person.
The preprocessing pipeline that actually helped
After several rounds of evaluation where the validation accuracy looked good but real-world performance was inconsistent, I rebuilt the preprocessing pipeline from scratch with a different objective: reduce everything that isn't the gesture itself.
The main changes were normalising image brightness and contrast before training, applying random augmentations during training (random rotation, horizontal flip, brightness jitter) to force the model to become invariant to the irrelevant stuff, and background removal or simplification on the training set where possible. These weren't novel techniques — they're standard practice. But understanding why they were necessary, not just that they're recommended, changed how I thought about every subsequent modelling decision.
The gap between benchmark accuracy and real-world performance
This is the part I want to be honest about, because I think a lot of portfolio projects gloss over it. My benchmark accuracy — measured on a held-out test set drawn from the same distribution as training — looks reasonable. My real-world performance, tested by showing the model live camera input, is meaningfully worse. Not broken, but not the same number.
Benchmark accuracy measures how well your model handles the test set. Real-world performance measures something different: how well it handles everything else.
This gap is the thing GestureKey is still working through. The current focus is on closing it — building a more diverse training set that better represents the range of conditions a real user would show the model, and iterating on evaluation criteria that are closer to the actual accessibility use case than standard accuracy metrics.
What I'd tell someone starting a similar project
Spend more time on your data than you think you need to. Not because architecture doesn't matter — it does — but because even the best architecture can't fix a training set that's teaching your model the wrong things. Evaluate on data that's genuinely different from what you trained on, as early and as often as possible. And if you're building for accessibility, the evaluation criteria that matter aren't the ones in the paper you're benchmarking against. They're the ones that reflect what real users actually need the system to do.
GestureKey is still in active development. The live browser demo uses MediaPipe for landmark detection, which sidesteps some of the raw image classification challenges — but the underlying research problem of building a system that generalises reliably across diverse inputs is the part I'm still working on, and the most interesting part of the project.