MIT engineers have developed a method that enables robots to make intuitive, task-relevant decisions in complex environments in the real world. The applications of the new method range from factory and domestic robots to search and rescue operations.
The new method is called Clio, like the Greek muse of history, for its ability to identify and remember only the things that really matter for a given task. For example, a quadruped robot (Spot by Boston Dynamics) running Clio on-board in real-time identified and mapped only those parts of an office building that related to the robot’s tasks (such as retrieving a dog toy while ignoring piles of office supplies), allowing the robot to grasp the objects of interest.
The Clio system is described in a research paper published in Robotics and Automation Letters. The final manuscript of the paper is available from MIT, and a preprint is published in arXiv.
With Clio, a robot is given a list of tasks described in natural language and, based on those tasks, it analyzes its surroundings to identify and remember only what is relevant to the given tasks.
“Search and rescue is the motivating application for this work, but Clio can also power domestic robots and robots working on a factory floor alongside humans,” says lead researcher Luca Carlone in a MIT press release. “It’s really about helping the robot understand the environment and what it has to remember in order to carry out its mission.”
Combining computer vision and LLMs
On-board Artificial Intelligence (AI) for robots that must navigate and operate in complex environments is important and critical. Such systems would be useful not only for search and rescue operations and the other applications mentioned in this study, but also for all applications that require robots to operate as autonomously as possible, such as law enforcement operations, military operations, and operations in deep space.
Clio makes use of both computer vision and large language models (LLMs), and can be seen as an advance toward multimodal AI.