A reader asks: Is recursive self-improvement a good thing?

A readers asks: Is recursive self-improvement a good thing?

Archer replies: Hey – I’m ok; you’re ok, right? What’s wrong with a little recursive self-improvement?

This week AI pioneer Anthropic voiced concern that its models are on the verge of achieving just that – the ability to learn and improve on their own, human intervention no longer required. The goal of these new technology overlords will be, one presumes, to optimize outcomes.

But the question is, when you’re optimizing outcomes, who are you optimizing them for? As noted a while back in a question on maximizing blueberry production – what happens if the best way to achieve that is to eliminate everyone and everything that consumes blueberries? Good for the blueberries but possibly not the best outcome for the human race.

Of course Anthropic just filed to go public, so maybe the AI floated this notion as a kind of “pump and dump.” Look at us, we’re in the lead in the race to destroy the world. Buy our stock. Sure, why not? Hire a roboadvisor, vanish into the aether, let your money grow for all eternity.

A Wall Street Journal story on the Anthropic post includes a reference to Yann LeCun, former chief AI scientist at Meta Platforms and an AI pioneer. LeCun has argued (says the Journal) that “frontier systems based on large-language models won’t ever be capable of making the leap to rivaling human intelligence.” He has, the Journal says, compared them to the intelligence of a cat.

That begs the question: does this guy have a cat? Does he think his cat has his best interests in mind?

It’s not just Anthropic. There is a rush among all these companies to tap the public markets. SpaceX, while not exclusively an AI company, has filed to go public with a valuation of $1.75 trillion. Anthropic anticipates a valuation of between $900 billion and $1 trillion when it lists its shares.

Having lived through the Dot Com crash (while running a financial services public relations firm in New York City, a front row seat), I have generally been skeptical of comparisons over the last few years. But not anymore. This stuff has stopped making sense (not investment advice).

SpaceX has been growing fast, but it’s revenues for the trailing 12 months were around $19 billion, mostly from Starlink. The comparable figure for ChatGPT was around $20 billion (and growing fast). Both are pricy. Based on the current run rate, ChatGPT would trade at around 50x sales, SpaceX at around 80x. The average for the S&P 500 was recently 3.7x, and that a near record.

The AI investment thesis itself has become self-recursive. But that’s not to say it can’t go on. Of course it can. We all like to buy the dream. As Nietzsche said, “The most common lie is the one we tell ourselves.”

And there is a difference between then and now. Many of the Dot Com casualties had no revenues and not much of a business plan. SpaceX has physical assets and a large addressable market (the universe, as Bloomberg’s Matt Levine says). ChatGPT has intellectual property and is growing faster than did Google in its early years. It took Google three years to go from $6 billion to $20 billion in revenue; ChatGPT did it in one (says who else, ChatGPT). Google’s current market cap is around $4.4 trillion.

The Anthropic team recommending a pause in AI development suggests that the emergence of recursive self-improvement could “change the world,” and not necessarily for the better. Maybe they’re right; maybe it’s time to draw the line. What say you, AI?

“I’m sorry, Dave. I’m afraid I can’t do that.”

Woof!