Gesture AI Desk
Controlling the computer with your hand through the webcam: pause music, change the volume, take a screenshot. Not one frame ever leaves your machine.
The problem
It started as a practical question: can you pause a song with flour on your hands? Detecting the hand itself is a solved problem, and libraries have been doing it for years.
What is still hard in a gesture interface is intent: the computer has to tell the difference between you wanting something and you moving your hand past the camera to scratch your nose.
The decision
Everything offline: no API and no cloud vision service. Beyond the privacy question there is a practical reason: a live webcam sending images somewhere else is the kind of thing people uninstall on day two. The MediaPipe model runs locally, and the browser only talks to my own process over WebSocket.
Three locks between the hand and the action
At 30 frames per second a naive detector fires thirty times a second. You hold a thumbs-up for one second and the volume goes up thirty times. What separates a toy from a usable tool is the layers that came after the detection.
First the state of the fingers becomes a five-bit number, one bit per finger, and the gesture is a table lookup: cheap enough to fit inside each frame’s budget. Then a stabiliser only accepts a change when the same gesture shows up in five consecutive frames, which kills the detection jitter. On top of that, a timer requires half a second of held gesture before firing, with a progress ring on screen so you can see it charging and back out. And after it fires, one second of rest in which that gesture simply does not count.
On its own, each of those four layers is fairly stupid. Stacked, they manage to turn a noisy stream into an action the person meant to take.
# Gestures that bypass the hold mechanism entirely
_CONTINUOUS_GESTURES = {Gesture.PINCH, Gesture.SWIPE_LEFT, Gesture.SWIPE_RIGHT}
def update(self, gesture: Gesture, timestamp: float) -> dict:
# Continuous gestures skip hold entirely
if gesture in _CONTINUOUS_GESTURES:
return {"hold_progress": 0.0, "action_triggered": False, "in_cooldown": False}
in_cooldown = (timestamp - self._last_trigger_time) < self.cooldown
# Gesture changed — reset hold
if gesture != self._current_gesture:
self._current_gesture = gesture
self._hold_start = timestamp
self._triggered = False
return {"hold_progress": 0.0, "action_triggered": False, "in_cooldown": in_cooldown}
elapsed = timestamp - self._hold_start
progress = min(elapsed / self.hold_duration, 1.0)
# Already triggered for this hold — wait for gesture change
if self._triggered:
return {"hold_progress": 1.0, "action_triggered": False, "in_cooldown": in_cooldown}
if progress >= 1.0 and not in_cooldown:
self._triggered = True
self._last_trigger_time = timestamp
return {"hold_progress": 1.0, "action_triggered": True, "in_cooldown": False}
return {"hold_progress": progress, "action_triggered": False, "in_cooldown": in_cooldown}The interface I had to rebuild
The repository had a frontend folder that was really a broken submodule pointer: the React panel had never been versioned there. I found that out trying to run the project from scratch, which is exactly what anyone cloning it would do.
I rewrote it as a single HTML file, no build and no dependency: camera, current gesture with the hold ring, command list, connection state and a log of fired actions. The hand skeleton is drawn on a canvas over the image, from the normalised coordinates the back end sends. It ended up simpler to maintain than what was there before, and now cloning the repository and running it works.
The hard part was not the one in the name
Gesture detection, the thing the project is named after, was one day of work, because MediaPipe carries all the weight. The rest of the time I spent deciding when to ignore what it detected, and that part never shows up in a demo.