Large language models (LLMs) create text one small piece at a time, a process that can be slow and use a lot of computing power, especially with bigger models. Speculative decoding is a technique that tries to speed things up by first guessing several possible next pieces of text and then checking them all at once. Many earlier versions of this technique either needed extra training of the model or did not work equally well on different kinds of computer hardware.
Researchers at the Japan Advanced Institute of Science and Technology have created UniSpec, a framework that speeds up the generation of text without any extra training and without changing the final answers. The system automatically chooses the best number of guesses for whatever hardware is being used, scores how reliable short sequences of text pieces appear to be, and builds a more useful set of candidate guesses based on those scores. The same researchers also released Multi-SpecBench, a new test set that covers seven languages and seven different kinds of writing tasks so that the method can be checked more thoroughly than English-only tests allow.
How the new approach works
In trials the method was run with two well-known model families on several types of graphics processors. It produced answers up to 2.6 times faster than other training-free speculative decoding methods while the output stayed identical to the ordinary slow generation process. Because UniSpec needs no retraining and adjusts itself to the hardware, it can be added to existing systems as a simple extra module. This could lower costs and improve speed for virtual assistants, customer support tools, multilingual translation, code generation, educational tutors, and large cloud services.
The current tests cover only seven languages and have not yet included languages with more complex word forms such as Arabic. The method also needs access to the model’s raw internal scores, which may not be available in some closed systems. Future work aims to cover more languages, handle changing hardware conditions, and test the approach in additional real settings. Over the coming years, methods that adapt to hardware and require no extra training may become a standard part of practical AI systems, helping make powerful language models more widely usable and less demanding on energy.
This research is published in Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics.