A screenless, AI-centric smartphone is on its way following the announcement of the Humane AI Pin on Friday morning, but the company’s demo shows exactly why you need to be careful implicitly trusting generative AI.

The Humane pin is a small computer you wear on your shirt or lapel. It has a camera, speaker, and microphone, but no screen.

You can opt to project a barebones interface onto your hand or a wall, but the designers want customers to look away from screens and rely instead on interfacing directly with AI services using voice commands.

“It’s a standalone device and software platform built from the ground up for AI,” said Humane co-founder Imran Chaudhri.

Humane’s product is initially only being released in the US for a price of US$699 plus a US$24 per month subscription fee.

That fee pays for an internet connection through Humane’s virtual mobile network – it is partnering with US telco T-Mobile – along with API calls to cloud services like OpenAI.

This is all a bid to keep the form factor down by offloading hardware requirements while also leaning on the explosion of cloud-based AI services that have popped up over the last year.

AI is the key to the Humane Pin, co-founder Bethany Bongiorno said in a pre-recorded demo.

“We don’t do apps. Humane’s OS runs AI experiences that are on-device and in the cloud,” she said.

“The OS understands what you need and picks the right AI in the moment”

But during the company’s demo, the AI made a factual error that is indicative of the current problem with trusting generative AI.

“When is the next eclipse and where is the best place to see it?” Chaudhri asked.

The pin correctly responds by saying the next solar eclipse is on 8 April, 2024.

It then confidently says the best places to see the eclipse are “Exmouth, Australia and East Timor”: two places that won’t see the eclipse which is passing over Mexico and much of the US.

Humane will reportedly update its video to show the AI giving the correct information.

Humane isn’t the first company to stumble at an AI demo (Google famously lost over $140 billion in market value after an announcement of its Bard search engine contained a factual error), but the fact-checking mistake points to why AI isn’t yet ready to be its own operating system.

Most interactions with your phone screen at least try to give you an option to understand the provenance of the information it delivers.

Humane’s Ai Pin gets in the way of your fact-checking by design thanks to the premise from which Humane built its device: that smartphone screens are harmful.

Former Apple executive José Benitez Cong reportedly came out of retirement to join Humane and told the New York Times that the device might alleviate the guilt he felt from working on the iPhone.

As a result of its foundational premise, Humane appears to have designed its device in a way that limits the ability for customers to control and decide how computers are used.

At one point in the demo, Chaudhri picks up a book and asks the Pin how much it costs, to which the computer says, “It’s $28 online”.

But there are hundreds of bookstores online, which one did the AI choose and why?

When he holds up a handful of almonds and asks how much protein they have, the AI provides a confident answer. But how did it know? Did it count them and guess or use some other metric?

Humane has struck a deal to bundle its service with music streaming company Tidal so that when you ask to play a song, that request goes through its service.

It’s not a stretch to imagine Humane partnering with a retailer like Amazon for its shopping feature, or a specific fitness company for its health features, all in the name of a frictionless user experience that may come at the cost of true consumer choice.