.png)
.png)
.png)
Could generative AI be on the right path to AGI?
Jan. 28, 2025. 7 mins. read.
Interactions
We tend to assume that current generative AI technology is not good enough for AGI. But what if it is almost good enough?
Some time ago Ben Goertzel reported that somebody told him: “For the camp that continues to claim that throwing massive amounts of compute at LLMs isn’t a reasonable path to AGI… I don’t know if that’s accurate anymore.”
Ben then argued to confirm that no, throwing more and more computing power at large language models (LLMs) isn’t a reasonable path to artificial general intelligence (AGI).
The overall flavor of Ben’s arguments is captured by his analogy with music: music generation models trained on music up to the year 1900 “will not invent progressive jazz, neoclassical metal, grunge, acid house, etc. etc. etc.” I’ll come back to this analogy.
I’ve long been persuaded that Ben is right on this point. However, I’ll play devil’s advocate and try to argue that, yes, LLM-like big data crunching could be all (or almost all) of general intelligence.
Could statistical data crunching be almost enough?
I asked this question on social media, including the SingularityNET forums: Perhaps language is all there is? Or most of what there is? What I mean is that we could consider all forms of interacting with the rest of the world as a generalized form of language, and perhaps a generalized type of transformer technology with suitable training input would reproduce all aspects of cognition just like LLMs reproduce language.
Predictably, I received very skeptical replies. Come on, really now. How can language be all there is?
Ben conceded that everything is a language in some sense, but argued that real AGI will need a new framework of which today transformer-based LLMs could well be a part, but only a small one. Ben makes this point in his last book: “the basic architecture and algorithmics underlying ChatGPT and all other modern deep-NN systems is totally incapable of general intelligence at the human level or beyond, by its basic nature,” he says. “Such neural networks could form part of an AGI, sure – but not the whole cognitive part.”
But perhaps we shouldn’t dismiss the possibility that, yes, a generalized form of language could be all, or most, of what there is to AGI. The possibly at least merits further thoughts.
I’m not expressing a conviction, but a hypothesis. Current generative AI is not good enough for AGI. But what if it is almost good enough?
Let’s go for a drive
The example that comes to my mind is driving. When I’m driving, I use decades of experience with the ‘language’ of driving in the street. This language is composed by tokens like steer right/left, speed up, slow down etc. My experience tells me which token to ‘utter’ next, even without formal rules. All drivers know that in certain situations one should slow down, even if one is not able to say exactly why. It is your hands and your feet that know, so to speak. It seems to follow that a suitable ‘LLM’ trained with millions of hours of street videos of people driving – maybe call it an ‘LDM’, a Large Driving Model – could drive pretty well.
Could generative AI methods enable an AI, trained on a very large repository of videos of people driving in all sorts of different situations, to drive a car well enough? I think this hypothesis is worth exploring.
OK, maybe driving. But AGI? Come on.
Well, of course AGI will need other things as well, for example reasoning models for logical thinking, inference, mathematics and that sort of things. And some good old Bayesian reasoning of course. But I think of those subsystems as a very thin surface layer on top of a very thick bulk.
Vectors in the mindscape
If you ask me if there is a largest prime number, I’ll answer that no, there isn’t, and I’ll tell you exactly how I have reached that conclusion. But ask me why I like this particular woman at first sight instead of liking that other woman? The only honest answer I can give is that I don’t know (or care), but I still like her at first sight.
Do you still want an intellectual answer? Well, I guess there must be some vector that represents this woman in the bulk of my hugely multidimensional mindscape, and the tip of the vector comes close to some ideal woman.
But isn’t this a good description of what LLMs do?
So I’m entertaining the idea that some kind of LLM on steroids could be a good model for most of my mind. I claim to be generally intelligent, so it follows that some kind of LLM on steroids could achieve general intelligence. Of course the LLM should be complemented and enhanced by other things. But it would be the major part of a general intelligence, not a small one.
So I do agree with Ben, but with a difference in emphasis. He sees the LLM glass half empty, but I see it half full.
Perhaps I should have said that today’s early LLMs are on the right path to reproduce not general intelligence (the surface layer), but the more primitive intelligence (the bulk) that enables us all animals to stay alive in the cold, unforgiving universe out there. But I suspect that the gulf between the two is not that wide.
Active inference
There are interesting parallels and analogies between LLMs and a theory of sentient behavior called active inference, originated by Karl Friston and other scientists. The theory suggests that sentient life forms act upon their environment to build and continuously refine an internal model of the environment.
This is not limited to sentient life but rather is “something that all creatures and particles do, in virtue of their existence,” suggests Friston. The theory is based upon a “free energy principle” that has been proposed to unify information, thermodynamics, and biology. “For Friston, the free energy principle explains all features of living systems,” notes Anil Seth, and is “as close to a ‘theory of everything’ in biology as has yet been proposed.”
The analogies suggest that, perhaps, today’s early LLMs manifest the same universal forces that produced you and me.
And what about consciousness?
Thomas Nagel, in ‘What is it like to be a bat?‘, said conscious exists when it’s “like that” to be a certain being. In that sens, I am persuaded that it will be “like something to be an LLM” (say, a future one called GPT 10, or perhaps even GPT 7), even if it’s very different from our familiar experiences.
Back to progressive jazz
Let’s go back to Ben’s music analogy. Train generative AI on music up to the year 1900, and run it at low temperature. The AI will produce decent imitations of the music styles on which you trained it, though probably a bit aseptic and unimaginative. But now raise the temperature to the point where the AI produces music on the edge of chaotic noise, which only vaguely reminds of its training set. Most of that music will be unpleasant noise that nobody wants to hear. But now and then a high temperature run will produce something that some listeners will find at least interesting. Those samples will be publicized, discussed in music books, and included in new training sets. So there will be a gradual drift toward new styles, and perhaps we will get progressive jazz and all that.
I think this non-linear feedback will soon be evident in language and literature. Much of the text that we read online is already written by AI, and much more will come. AI started imitating people, but soon people will start imitating AI and picking up some new expressions produced by AI.
To be continued…
I’ve been writing this article for months. Of course there’s a lot more to be said, but I didn’t want to wait forever. So I’ll continue thinking and write a follow-up soon. In the meantime, there are always interesting discussions on X, like this one where Ben and I participated.
Let us know your thoughts! Sign up for a Mindplex account now, join our Telegram, or follow us on Twitter.
0 Comments
0 thoughts on “Could generative AI be on the right path to AGI?”