;)
Speak to the hand!
Posted by Bridget in Physics & Maths
Researchers make gesture-based computing interfaces more accessible with single piece of inexpensive hardware and multicoloured glove.
By Larry Hardesty, MIT News Office
Academic and industry labs have developed a host of prototype gesture interfaces, ranging from room-sized systems with multiple cameras to detectors built into laptops’ screens. However, MIT researchers have developed a system that could make gestural interfaces much more practical. Aside from a standard webcam, like those found in many new computers, the system uses only a single piece of hardware: a multicoloured Lycra glove that could be manufactured for about a dollar.
Other prototypes of low-cost gestural interfaces have used reflective or coloured tape attached to the fingertips, but ‘that’s 2-D information,’ says Robert Wang, a graduate student in the Computer Science and Artificial Intelligence Laboratory who developed the new system together with Jovan Popović, an associate professor of electrical engineering and computer science. ‘You are only getting the fingertips; you do not even know which fingertip [the tape] is corresponding to.’ Wang and Popović’s system, by contrast, can translate gestures made with a gloved hand into the corresponding gestures of a 3D model of the hand on screen, with almost no lag time. ‘This actually gets the 3D configuration of your hand and your fingers,’ Wang says. ‘We get how your fingers are flexing.’
It’s all in the game
The most obvious application of the technology, Wang says, would be in video games: Gamers navigating a virtual world could pick up and wield objects simply by using hand gestures. However, Wang also imagines that engineers and designers could use the system to more easily and intuitively manipulate 3D models of commercial products or large civic structures.
The glove went through a series of designs, with dots and patches of different shapes and colours, but the current version is covered with 20 irregularly shaped patches that use 10 colours. The number of colours had to be restricted so that the system could reliably distinguish the colours from each other, and from those of background objects, under a range of different lighting conditions. The arrangement and shapes of the patches were chosen so that the front and back of the hand would be distinct but also so that collisions of similar-collared patches would be rare. For instance, Wang explains, the colours on the tips of the fingers could be repeated on the back of the hand, but not on the front, since the fingers would frequently be flexing and closing in front of the palm.
Technically, the other key to the system is a new algorithm for rapidly looking up visual data in a database, which Wang says was inspired by the recent work of Antonio Torralba, the Esther and Harold E. Edgerton Associate Professor of Electrical Engineering and Computer Science in MIT’s Department of Electrical Engineering and Computer Science and a member of CSAIL. Once a webcam has captured an image of the glove, Wang’s software crops out the background, so that the glove alone is superimposed upon a white background. Then the software drastically reduces the resolution of the cropped image, to only 40 pixels by 40 pixels. Finally, it searches through a database containing myriad 40-by-40 digital models of a hand, clad in the distinctive glove, in a range of different positions. Once it has found a match, it simply looks up the corresponding hand position. Since the system does not have to calculate the relative positions of the fingers, palm, and back of the hand on the fly, it is able to provide an answer in a fraction of a second.
One size fits (almost) all
Of course, a database of 40-by-40 colour images takes up a large amount of memory – several hundred megabytes, Wang says. However, today, a run-of-the-mill desktop computer has four gigabytes – or 4000 megabytes – of high-speed RAM memory. Moreover, that number is only going to increase, Wang says.
Since the glove is made from stretchy Lycra, it can change size significantly from one user to the next; but in order to gauge the glove’s distance from the camera, the system has to have a good sense of its size. To calibrate the system, the user simply places a 215x280mm piece of paper on a flat surface in front of the webcam, presses his or her hand against it, and in about three seconds, the system is calibrated.
Wang initially presented the glove-tracking system at last year’s Siggraph, the premier conference on computer graphics. However, at the time, he says, the system took nearly a half-hour to calibrate, and it did not work nearly as well in environments with a lot of light. Now that the glove tracking is working well, however, he is expanding on the idea, with the design of similarly patterned shirts that can be used to capture information about whole-body motion. Such systems are already commonly used to evaluate athletes’ form or to convert actors’ live performances into digital animations, but a system based on Wang and Popović’s technique could prove dramatically cheaper and easier to use.
Source: MIT
Caption: The palms of the Lycra gloves. (Image: Jason Dorfman/CSAIL)
