It's now summer of 2028 and one of the major AI powerhouses makes a breathtaking announcement: they've achieved Super AGI. Far superior to any human, this AI answers all your questions far better than you ever could. The company assures you, you can rest confidently in the knowledge that this AI is reliable, because it's more powerful than any human in history. It's Super-AGI!

With this AI, you might consider having it:

• diagnose and fix your car
• write up your will without hiring an attorney
• invest your retirement savings
• cook dinner including wild mushrooms identified by the AI as non-toxic
• design and build your own personal aircraft

Would you use AI to do any or all of these things? And here's the more important question: would you trust its answers implicitly? Or would you double-check?

I think most rational people would double-check at least some of these scenarios. But why? It's Super AGI! It's mathematically proven to be smarter than you! Why don't you just...trust it?

This is the principle of verification and it's why we don't just release new technology into the wild without testing it first. Technology is complicated, hard to reason through, and easy to err. As any curmudgeonly engineer will tell you, it's not about the implementation, it's about the verification. We all know stories about what goes wrong when technology hasn't been verified properly.

AI is different than previous technological evolutions, simply because it is so difficult to verify. Machine learning is stochastic (random) by nature. This leads to lack of repeatability and precision, two characteristics that are the hallmarks of high-quality engineering and verification.

This in turn presents challenges when adopting AI in the workplace. The automation potential is huge, but this automation differs from previous automation. In the old world, we experienced highly consistent results. In this new world of automation, anything above 90% is a home run.

What about vibe coding, you may ask? Software coding requires extreme precision and cannot tolerate any failures whatsoever. If AI fails so often, how is it able to automatically write so much code?

As it turns out, coding is probably the best use-case for AI assistance, it's "killer app". Coding is an entirely digital experience, with limited cross-over into the real world. This allows the AI to self-verify. Any skilled coder using AI assistance will tell you that your results improve dramatically when you set the AI up with automatic verification. This provides feedback loops that allow the AI to change its answers based on whether a task succeeds or fails. Code doesn't compile? Edit, try again. Test failing? Edit, try again. It can self-improve, because it can self-verify. And this self-verification allows the technology to mask the fact that it's actually failing all the time, it just recovers gracefully. (Which ironically is exactly how humans do it too - there is nothing more humbling than the incredibly dumb mistakes one makes on a daily basis when coding.)

This coding AI success seductively leads people into believing that AI is able to substitute for humans, or at least it will be able to soon. After all, software coding is an incredibly difficult task - complicated and sophisticated. Coders are smart! If AI can replace coders, it can replace anyone, the thought goes.

But that perspective skips right over a very important question: when a task crosses over into the real world, how does AI self-verify? How do you self-verify a mechanical defect on an oil rig in the North Sea? How do you self-verify an international diplomacy proposal? A legal brief for a judge? So much of what we do in the world isn't testable at all, much less confined to a purely digital environment.

Now you may be wondering, "well what can we use AI for?" And my answer may surprise you: pretty much anything! You just need to know how you're going to verify. And the easiest way to do that is let the human be the verification. Use AI as an augmentation rather than a replacement. A practical discussion of that is coming up in my next essay.