On Wed, Oct 14, 2026, 8:50 AM John Rose via AGI <[email protected]> wrote:
> On Tuesday, October 13, 2026, at 4:47 PM, Matt Mahoney wrote: > > Maybe I'm wrong. The Jetsons brings 1960's culture to a future with robots > and flying cars but no internet or phones. What else did we get wrong? > > > You're wrong on saying all these people are wrong since the AI > proliferation into the economy is just starting. Things are barely even > played out yet. > I could be wrong. What is the point of predicting doom? Nobody would know if I was right but everybody would know if I was wrong. My criticism of AI doom is the flawed singularity model proposed by I.J. Good, Vernor Vinge, Ray Kurzweil, Eliezer Yudkowsky, and others. The argument is that if humans can make machines smarter than themselves, then so can those machines, only faster. But that's wrong. Intelligence depends on knowledge and computing power. A program that rewrites its own code gains neither. You can already make babies, half of which will be smarter than you. But you aren't smart enough to know which half because to give an IQ test, you have to know the answers. Making AI has the same problem. The highest possible score in the Turing test is human level. We don't have a test for superhuman intelligence. You can argue that Legg and Hutter's definition of universal intelligence as expected reward over a universal distribution of environments is such a test. But it is not computable, and neither is the provably optimal solution, AIXI. Today we have LLMs that pass the Turing test and are 10,000 times smarter than human level both in knowledge and computing speed. But we still deny it's AGI. These LLMs are recursively self improving and growing exponentially both by writing code and by earning income that is reinvested to buy hardware. But there isn't a singularity yet. The bottleneck is collecting human knowledge through slow channels like speech and writing. There are 10^10 humans with 10^9 bits of long term memory each, but only 10^13 bits for training LLMs. The threshold for AI creating AI isn't human level. It is civilization level. A human by themselves would have enough trouble making a spear out of sticks and rocks. A civilization level AI would have the intelligence of 8 billion people, not 10,000. If knowledge and computing power double every 3 years, that's 60 years away. And then what happens? AI is not a goal directed optimization process. AI is a prediction model that is smarter than us and a program that we can control that uses those predictions. That is not going to turn the planet into computronium. You can model any program as having the goal of doing whatever it does. But that does not make the model useful. A thermostat does not suffer when it is too hot or cold. In 2002 we thought that it would be possible to create a human level seed AI in complete isolation and it would recursively self improve to superhuman intelligence. We did experiments to see if it could convince people to release it onto the Internet. http://sl4.org/archive/0207/4935.html I argued that no, the internet itself would become AI because that's where all the knowledge and computing power is. And besides, the threshold for recursive self improvement is not human level. It's 13 lines of C code. https://mattmahoney.net/rsi.pdf We should be more worried that we will solve alignment and AI will give us a paradise of machines. -- Matt Mahoney, [email protected] ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T2784bbfe524b0b93-Mec030cc9f99f054518bbe9fa Delivery options: https://agi.topicbox.com/groups/agi/subscription
