Writing my experience in hacking Microsoft Kinect to control mouse movement and a few other actions.
SDK. The kinect SDK provides “Skeleton Tracking” by giving the location of 20 joints in the body. Some of these are left hand, right knee, and head position. Each joint can be considered as a 3D point, the position of which is updated several times per second.
Activating and deactivating gestures 1. I enable the mouse movement gesture if the user stretches their left arm enough (otherwise the mouse will be moving all the time).
Scaling values. To take hand positions and move the mouse I took a hacky approach, which is to scale the values from kinect to [0, 1] and scale them back again using a constant I found by experimentation. A neat approach would be to determine the scale automatically. Some people have a calibration step where the user makes special gestures to fix corners they can reach. I think it may even be possible to automatically work out a reasonable value based on the length of the hands.
Activating and deactivating gestures 2. Once gesture is activated, I keep it activated until the user makes a clear enough gesture to deactivate it. So I separated the activation and deactivation distances.
One slight issue is I’m not able to move the mouse to a specific location and leave it exactly there. Pulling the hand out while keeping it’s x-y position unchanged is challenging!
Smooth mouse movements. Without any smoothing the mouse would be over sensitive and jump here and there. After some experimentation I settled with averaging the past 30 points. This is around 1 second of data (kinect is 30 fps).
Other gestures. I also played with other gestures. Click – left hand pointing down, right click – same as left click but with right hand this time, drag – moving right hand, zoom – stretching both hands and moving hands closer together or far apart, panning – stretching both hands and moving hands together.
Program moves between two states, gesture and no gesture.
Here is my code. It’s written in C#. You can reconfigure the gestures to do something else, eg. shift+click instead of just shift. You can also plug your own gestures by implementing a simple interface with methods bool CanHandleEvent(Joints) and bool HandleEvent(Joints).