MIT's new robot identifies things by sight and touch
The model is built on predictive AI that allows it to link multiple senses in much the same way humans doEuropost
For humans, it's easy to predict how an object will feel by looking at it or tell what an object looks like by touching it, but this represents a big challenge for machines. Yet, a new robot developed by MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) might have changed that.
In an effort to bridge the gap between humans' and robots' perception, the MIT researchers developed a predictive AI capable of learning to “see by touching” and “feel by seeing,” a means of linking senses of sight and touch in future robots. Using a KUKA robot arm with a tactile sensor called GelSight (another MIT creation), the team recorded nearly 200 objects with a web cam. These included tools, fabrics, household products and other every day materials humans come into contact with regularly. The team then used the robotic arm to touch the items more than 12,000 times, breaking each of these video clips into static frames for further analysis. All told, researchers ended up with more than 3 million visual/tactile paired images in its dataset.
“By looking at the scene, our model can imagine the feeling of touching a flat surface or a sharp edge,” said Yunzhu Li, CSAIL PhD student and lead author on the paper. “By blindly touching around, our model can predict the interaction with the environment purely from tactile feelings.”
According to Li, the successful incorporation of the two senses together could empower the robot and reduce the data needed for tasks involving manipulating and grasping objects.