Recent research indicates that adding human-like memory restrictions to artificial intelligence (AI) can improve how these systems acquire language. Researchewrs carried out the study using small language models. The researchers gave the models a transient memory, which means the systems gradually lose access to older details about words and sentences.
This approach draws from cognitive science. For many years, experts in this area have argued that human memory limits support language learning. People quickly forget exact word forms while processing speech or text. This forgetting may push learners to notice recurring patterns and build general rules of grammar instead of relying on specific examples.
Memory decay
The researchers tested the idea by modifying common transformer language models. They added memory decay together with a short echoic memory buffer. The buffer kept only the most recent three to seven words available at any moment. The models were trained on the BabyLM benchmark, a data collection meant to match the amount of language a child typically hears while learning to speak. This allowed a clear comparison between models with and without the memory features under conditions similar to human learning.
Models that used memory decay showed better performance in basic language tasks. They also developed stronger understanding of sentence structure and grammar rules. The improvements appeared consistently only when memory decay was combined with the short buffer. Models without both elements did not show the same gains.
An unexpected result emerged in further checks. Although the models learned language more effectively, they became less accurate at predicting human reading times. These predictions are usually based on measuring how surprising a word is given the preceding context. The researchers noted that current explanations do not fully account for why better language learning did not lead to better matches with human reading behavior.
The study supports the view that memory limits can aid language acquisition even in modern artificial systems. It also suggests that the factors helping models learn language well may differ from those that best predict how humans process language moment by moment. The work shows how ideas from human cognition research can guide new methods for training AI.
This research is published in Computational Linguistics.