A new method helps robots become better assistants by understanding both what people say and how they point. Researchers at Brown University created this approach so robots can fetch the right objects even when rooms are messy or items are partly hidden. In tests the system reached an 89 percent success rate clearly better than methods using only words or only gestures.
The method is built around a mathematical tool called a POMDP which stands for partially observable Markov decision process. This framework lets the robot make decisions even when it is unsure about its surroundings by keeping track of probabilities and updating its beliefs as it gathers new information.
How gestures and vision models work together
To understand pointing the researchers studied how dogs interpret human gestures since dogs are very good at following points. They modeled a pointing gesture as a cone-shaped area of probability based on the line from a person's eye through their elbow to their wrist. This model was then combined with a vision language model. A vision language model is an artificial intelligence (AI) system that can understand both pictures and natural spoken language at the same time.
In laboratory experiments a four-legged robot was asked to find different objects in a room. When people gave spoken instructions and pointed the robot moved around checked different angles and successfully located the correct item almost nine out of ten times. The approach works well because it lets the robot handle uncertainty instead of making quick mistakes.
The research brings robots closer to working naturally alongside people at home or in workplaces. Future versions could include more types of human communication such as eye movements and demonstrations. This combination of computer science and studies of how humans and dogs interact shows a promising direction for more intuitive robot assistants.
The researchers have described the methods and results of this study in a preprint published in arXiv.