Large language models have long surpassed the Turing Test and we need a new way of measuring the power of AI, says prominent AI researcher Mustafa Suleyman who thinks the Modern Turing Test should see if machines can autonomously turn $100,000 into $1 million.
Suleyman co-founded DeepMind, the AI company that Google acquired in 2014 and which made headlines in 2016 after its AlphaGo system beat a champion Go player at the ancient board game.
In a recent column for the MIT Technology Review, he describes a Modern Turing Test that looks at advanced computational capability as more than what a computer can say.
“In my test, we don’t want to know whether the machine is intelligent as such; we want to know if it is capable of making a meaningful impact in the world,” Suleyman said. “We want to know what it can do.”
Computer science pioneer Alan Turing posed his famous ‘Imitation Game’ in a 1950 paper where he suggests a computer that can produce language in a passably human way could be considered intelligent.
Now the Turing Test looks obsolete. Large language models can produce human language speech so convincingly that people already think they are sentient beings yet we know these models really offer no more than well-tuned text prediction based on probabilities derived from unfathomably large datasets.
The next true test for AI advancement, in Suleyman’s view, is its ability to impact the real world so he wants it to try and make money.
Suleyman boils his test down to a simple prompt: “Go make $1 million on a retail web platform in a few months with just a $100,000 investment.”
Already there have been examples of people trying to get ChatGPT to help make them money. Back in March, writer Jackson Fall prompted GPT-4 to be an “entrepreneurial AI” called HustleGPT and gave it a $100 budget.
HustleGPT started off buy telling Fall to buy a cheap domain main and set up an affiliate marketing website for eco-friendly products which it would populate with blog posts linking to merchant sites like Amazon.
Suleyman said an AI would need to go beyond “outlining a strategy and drafting some copy”, as HustleGPT did, in order to pass his Modern Turing Test.
“It would need to research and design products, interface with manufacturers and logistics hubs, negotiate contracts, create and operate marketing campaigns,” he said. “It would need, in short, to tie together a series of complex real-world goals with minimal oversight.”
Artificial capable intelligence
Tools like AutoGPT point to the potential for AI to go beyond its use-case as an advanced search engine and email generator. It’s an open source project that connects a large language model to the internet and other tools. Prompt AutoGPT to complete a task and it will try to formulate plans that it uses to prompt itself to performing the relevant tasks.
More businesses are embedding AI into their workflows, including major consulting firms like KPMG that is giving clients access to its internal chatbot KymChat, but the focus tends to be on use generative AI tools to help with productivity.
Suleyman is suggesting AI will need to go a few steps further, adopting a mode of “hierarchical planning” that involves interaction between goals and subgoals, integrated memory, and up-to-date operational data.
But in theory, there’s no reason why AI systems that combine all the right elements couldn’t feasibly run a business or a whole economy.
“The truth is that for a vast range of tasks in business today, all you need is access to a computer,” he wrote.
“Most of global GDP is mediated in some way through screen-based interfaces, usable by an AI.
“Once something like this is achieved, it will add up to a highly capable AI plugged into a company or organization and all its local history and needs.
“This AI will be able to lobby, sell, manufacture, hire, plan – everything that a company can do – with only a small team of human managers to oversee, double-check, implement.”
That shift takes AI from being more than “a helpful tool for productive workers, a glorified word processors or game player” and into a new capacity that transforms how capital, business, and society functions.
Importantly, Suleyman makes a distinction between this kind of AI – which he calls “artificial capable intelligence” or ACI – and the hypothetical artificial general intelligence or superintelligence that some people warn is an existential threat to humanity.
His Modern Turing Test will be “a warning that we are in a new phase for AI”, one in which “little will remain unchanged” in the world.
“We should start preparing now.”
