Researchers from The Hong Kong Polytechnic University, Ohio State University, Princeton University, and City University of New York have explored how large language models form ideas about words compared to humans. The study suggests that sensory input is key.
The researchers compared word ratings from humans and language models like ChatGPT and Google’s models. They used a dataset of about 4,500 words, rated for qualities like concreteness (how tangible a concept is), imageability (how easily it forms a mental picture), and sensory traits like smell or sound. They also looked at motor traits, like actions involving legs or mouth.
By comparing human and model ratings, they measured how similar their understanding of words was.
Sensory input and model performance
The study found that language models struggle more with sensory concepts, like colors or smells, and even more with motor concepts, like running, which rely on physical actions. This suggests text alone isn’t enough for models to fully understand concepts tied to senses or movement. To test if sensory input helps, the researchers compared models trained only on text, like GPT-3.5, to those trained on text and images, like GPT-4. Models with visual input were much closer to human understanding, showing that adding sensory data improves their grasp of concepts.
The findings align with human studies showing that sensory experiences, like seeing and touching, shape how we understand objects. For language models, combining text with images helps them form richer concepts, similar to how humans learn. In the future, equipping models with more sensory inputs, like through robots that interact with the world, could make their understanding even closer to ours. This could lead to models that capture the full richness of human concepts.
The researchers have described the methods and results of this study in a paper published n Nature Human Behaviour.