The year for on-device AI

Humane and Rabbit have been the most talked-about tech hypes of the last month. They’re utterly disappointing.

The utopia of a device that ‘does it all’ just by talking to it seems out of reach. Why? These devices still need to connect to a costly data center to process voice commands. This brings an awkward delay, making it painfully clear we’re dealing with a machine. The illusion shatters when the answer is completely wrong. And it’s even worse when the battery dies just when you need it most.

From a business perspective, it’s clear that using a generative large language model from the cloud is economically unviable.

But I think we’re closer than we realise.

There’s been a surge in open-source development of models that run on various sizes of chips. This progress is largely invisible.

A user-friendly interface like ChatGPT can be deployed and shipped to millions of users overnight. Open-source models, however, are less intuitive, often requiring a tedious installation process. Even then, they can only be used from the terminal.

But the underlying power should not be underestimated. If we can standardise these models and run them locally, we can build an interface for them. I think we’re closer than we think. Google’s model is designed to run on phones, and the A-chips in our iPhones are capable of similar feats.

The trick with AI is to create a seamless blend of technologies for the user. A good design and user experience that leverages a combination of traditional algorithms and generative models.