MIT's new robot identifies things by sight and touch

The model is built on predictive AI that allows it to link multiple senses in much the same way humans do

For humans, it's easy to predict how an object will feel by looking at it or tell what an object looks like by touching it, but this represents a big challenge for machines. Yet, a new robot developed by MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) might have changed that.

In an effort to bridge the gap between humans' and robots' perception, the MIT researchers developed a predictive AI capable of learning to “see by touching” and “feel by seeing,” a means of linking senses of sight and touch in future robots. Using a KUKA robot arm with a tactile sensor called GelSight (another MIT creation), the team recorded nearly 200 objects with a web cam. These included tools, fabrics, household products and other every day materials humans come into contact with regularly. The team then used the robotic arm to touch the items more than 12,000 times, breaking each of these video clips into static frames for further analysis. All told, researchers ended up with more than 3 million visual/tactile paired images in its dataset.

“By looking at the scene, our model can imagine the feeling of touching a flat surface or a sharp edge,” said Yunzhu Li, CSAIL PhD student and lead author on the paper. “By blindly touching around, our model can predict the interaction with the environment purely from tactile feelings.”

According to Li, the successful incorporation of the two senses together could empower the robot and reduce the data needed for tasks involving manipulating and grasping objects.

Similar articles

  • iPhone 13 targets satellite connectivity

    iPhone 13 targets satellite connectivity

    Apple is just about to unveil a revolutionary model of its iPhone. If the technological conglomerate keeps its schedule for introducing new models, the audience may see it in just a few weeks with the premiere of the iPhone 13. Strong signals suggest that the new edition of the Apple flagship could have a built in potential for satellite connectivity, The Verge reported.

  • Google bets on Samsung 5G modem for US market

    Google bets on Samsung 5G modem for US market

    The technological conglomerate Google bets on Samsumg to bring up a 5G modem for its next flagship smartphone Pixel. This will be the first win for the Korean firm in a US market dominated by Qualcomm, Reuters reported. Earlier in August Google unveiled that it had designed its own processor chip to power its new Pixel 6 and Pixel 6 Pro high profile phones, ending its complete reliance on Qualcomm, which would keep supplying chips for the cheaper Pixel 5A.