Humans easily recall where they left an item the day before and return to that exact spot. Robots working alongside people often lack this ability to store and retrieve memories that combine space and time. Researchers at MIT have created a new memory system to address this gap. The framework lets a robot build a rich mental model of large environments as it moves through them over long periods. It stores both three dimensional maps of locations and detailed descriptions of objects seen along the way.
The system combines two existing approaches in useful ways. One approach uses computer vision to describe objects in detail. The other builds three dimensional maps of entire spaces such as buildings or campuses. Earlier methods either lacked rich descriptions or ran too slowly for practical use on moving robots. The new framework solves these limits by grouping nearby objects into regions on the map and selecting only the clearest images for detailed description. This speeds up the process by about ten times while still capturing useful information about each object.
How the new memory system works
Once the map with descriptions is built, the robot stores it in a large database. To answer a question, the system uses a large language model together with special tools that pull out relevant facts quickly. This method reduces errors where the model might otherwise invent incorrect details. Tests showed the framework answered questions 21 to 53 percent more accurately than previous approaches, depending on the type of question asked. It also responds in just a few seconds, which is fast enough for a moving robot to use during real tasks.
The memory system could support factory robots and augmented reality tools that help maintenance workers spot problems or guide people through complex buildings. The long term goal is to create robots that can handle a wide range of helpful tasks by understanding space, time, and language in ways that match human reasoning.
This research was presented at the Conference on Computer Vision and Pattern Recognition.