Fallbacks will be the death of us

In software development, a fallback is a catch-all, a way to resort to a less-good-but-still-works way of doing things. Your car radio is a fallback to streaming if you don't have Internet. Boiling water is a fallback to a water treatment plant.

And in my work building things with AI, fallbacks have become my mortal enemy.

Agentic engineering is fundamentally different from an era when code was expensive and precious. We used to conduct careful designs, laying everything out in happy paths and error messages. We'd iterate in Figma, do paper prototypes, add our taste and flourish, and make sure everything worked in our heads before it worked in code.

Because code was expensive.

The wrappers around the coding process—customer research and design before, QA and testing after—existed because we needed to keep the part in the middle efficient. Now that code is cheap, it's altering the surrounding pieces in ways we don't yet fully understand:

• We build something, then see if it's usable. Instead of working out usability first, we build the MVP from a set of instructions. "Make a game of Battleship for two players" is a simple prompt that can realistically deliver a functioning app. But is it fun? We find that out when we actually play it.
• Our tests are optimizing for passing the tests. Goodhart's law says that once a metric becomes a target, it ceases to be a useful metric. But when you're building with AI, "does it pass the tests?" is a proxy for "does it work." There are many ways to make something pass a test: you can make the test easier, or give the thing you're testing the correct answers. In agentic engineering, this means the tests become dishonest.
• We add the taste later. We start with functionality, asking can we do the thing, then streamline it to decide how the thing should be done. (A side consequence of it being easy to build things is you have to be better at deleting them when they aren't actually good.)

Plenty of people are talking about how the design > build > test cycle is now build > test > design. Fewer mention the dirty secret: the tests are fiction and the design is the result of many iterations, each of which leaves a messy trail of how things used to be.

I've seen this many times in a project I'm working on. I'm deliberately obfuscating some of the details to avoid giving things away, but:

Tests are fiction

My test harness became dishonest:

• The thing I'm building requires that an AI agent complete a task solely based on a skill we give it. Things were working well. Too well. Then I found out that the AIs that built the test had been giving the agent hints to ensure its success. They were trying to pass the test, not test the desired behavior. This happens all the time. Your tests are lying to you.
• When you build test-driven software, you don't just write code, you write code to test that the code works. My project has well over 700 such tests. But deployment still fails because the tests are just rubber-stamping the functionality.

In other words: AI developers bend the test to suit the app until the app seems to work, instead of making the test represent the desired outcome and modifying the app until it works.

Iterations build up

I've experimented with many approaches to get to where I am are with this app.

The chatroom that wouldn't die

The nature of the project (yes, I know, being coy here) involves agents from Anthropic, OpenAI, and Google. So I'm using all three companies' AI tools (Claude Code, Codex, and Gemini) to build it. Rather than me acting as a go-between, I had the AI coders build a small messaging platform (think IRC). They called it Signalfire (don't judge.)

Shortly after, I realized I needed a task tracking app. Rather than use a tool like Asana to manage my Kanban Board, I had the AI coders build this too—and wire it into what was happening behind the scenes. I'm glad I did; there are, as of this morning, 525 tasks.

I asked the AI coders to integrate Signalfire (the chat app) into the task tracking tool. Which they did. The problem is they refused to excise the old app. Sure, they moved it into a directory called "defunct". But it kept coming up; they'd try to use the old thing, find out it didn't work, then use the new thing. The intended path relied on a fallback. Making the mistake was a required step in using the thing.

Your code refuses to forget

Some of the most promising features in this app started out very differently from where they are now. This is largely because of the build > test > design cycle I mentioned above. I didn't have a fully-formed understanding of what these things were when I started building them, because seeing the prototype was part of the process.

To be clear: this is amazing; it's reminiscent of Henry Mintzberg's Crafting Strategy: What you make emerges from the clay as it spins on the wheel.

But the early experiments were full of ideas that changed. They had placeholders and shortcuts. The goal was to get a stable, testable first version that humans could try, because there is no substitute for actually using the damned thing.

And as the AI coders built the new versions, they left the old ones in place just in case, in the form of fallbacks.

Initially this sort of behavior is done with the best of intentions. It's a belt-and-suspenders form of redundancy. Coding patterns like catch-if handle errors you didn't expect. Leaving in a "fall back" mechanism seems smart—but you shouldn't be falling in the first place.

These fallbacks all try to keep users on the "happy path". But in doing so, they mask that path's many hidden, unintended exits. Instead of a clear roadway that is a delight to navigate, you wind up with bunch of confusing, ambiguous directions and off-ramps to nowhere.

My job is now brain surgery

This happens so much in AI design that the agentic developers rely on a wide range of tools to fight it: The /simplify function; Superpowers; and increasingly frustrated human prompts that tell the AI coders to burn more tokens cleaning up their messes, or to check another AI coder's work in order to get back to the sort of clear, parsimonious code humans used to produce when coding was expensive.

I literally have one AI coder fixing another's memory at the moment.

The true name of the thing

The prompts themselves turn into prose and nuance. Ursula K. Le Guin was right: A thing's true name gives you power over it. In agentic engineering, words are spells that drag an LLM into the right "headspace" (though that headspace is actually a multidimensional embedding.)

I don't just write "clean up the code", I often write in analogies. At one point, the AI agents were building a feature (call it "milkshake" for the sake of this discussion). I realized that because of how it worked, it wouldn't scale. My instructions weren't simply "remove the milkshake." Instead:

"You are a skilled neurosurgeon working with precision tools and the finest surgical equipment, and today you need to remove the milkshake. It has become a cancer, a glioblastoma, and you must surgically remove every molecule of it so it doesn't metastasize. Excise it like a surgeon, inspecting every file, function, and piece of documentation to ensure it is completely removed. The patient's life is at stake here, so you must be precise and meticulous. Remove the milkshake completely."

That's not a prompt, it's a declaration of medical liability. But it's the kind of thing I resort to because the AI coders are focused on their objective functions rather than real outcomes. In their eagerness to build code that passes tests, they're leaving fallbacks all over the place that slowly turn into bloated, byzantine leftovers and half-forgotten off-ramps, instead of removing them.

The canary stopped singing

Sometimes, fallbacks linger because someone benefits from the complexity they create. Legal contracts and government regulations are prime examples of this:

• A contract grows to accommodate more and more "standard clauses." The people who sign the contracts don't really understand them, but dismiss them as "the way things are done." (I wrote about this for FWD50, a conference I chair, a while back.) These clauses are fallbacks. They catch rare exceptions, but their existence masks the underlying problems of the agreement itself. Clauses persist, when instead the contract could be cleaned up and simplified, because lawyers get paid by the line.
• Regulations expand to plug loopholes in outdated or badly-written laws. A regulation is supposed to enforce the intent of the law, but as the world changes, those regulations grow increasingly out of date. But lobbyists, bureaucrats, and public servants earn a living from finding, exploiting, and fixing loopholes, and politicians pay a cost for trying to change things, so the legal precedents build up while the law stays outdated.

But for most work, friction was the garbage collector. Expensive code got pruned because writing it was costly. Slow contracts got renegotiated when the pain became unbearable. Bad processes got redesigned when the workaround finally broke. Friction forced deletion, particularly in domains where there was an objective way to measure better or worse.

AI removes that friction, and in so doing, removes the garbage collector. When everything is cheap to add, nothing is cheap enough to justify removing.

I'm not opposed to fallbacks as a temporary tool. A fallback that you know about and control is just a contingency plan, and there's a time for it. But when your fallbacks become invisible load-bearing walls, they're debt that forgot it was debt. Fallbacks are the symptom; the disease is that deletion feels riskier than retention, and without friction, things stay. Your AI is letting Kato Kaelin crash in the guest house.

Simon Willison thinks we're headed towards a world of dark factories: fully-automated software development processes in which the agent can write, refactor, and test software without human intervention. They're serving new customers. Every kind of white-collar work is getting a factory.

And I don't trust the workers.

This is what AI governance actually looks like in business. It's not just bias and safety regulations (though those matter a lot.) It's the unglamorous work of building institutions, teams, and habits that can delete things and see when an AI is gaslighting them. You need the power to say: This is vestigial. It is debt that forgot it was debt. It is a way to cheat on the test. It masks a real problem. And it has to go.

The canary isn't singing anymore, because an AI moved it to /defunct.