How do I know an AI coding agent really fixed something?

My AI coding agent once told me it tested a feature. It didn't work. When I asked, didn't you say you tested it? It said yes, but not the right way.

"I don't trust you. You have to go take screenshots that you used it as a human to prove that."

Why can't I take the agent's word for it?

The biggest problem with Claude and Codex, in my experience, is claiming things work that don't because they didn't test them. Four failure patterns I've hit again and again:

1. Fake testing. It says it tested, but it tested something adjacent, or in a way no user would.
2. Local isn't live. My coding agent fixes things on my machine without flagging whether the fix is deployed, and doesn't explain tradeoffs in terms a non-technical operator can use. GitHub showing a merge doesn't prove production updated either.
3. "Good to go" without the downstream proof. I objected once when an assistant called a provider integration good to go without proof the automation it was supposed to trigger actually fired.
4. All the details, none of the point. It builds every supporting piece and never connects them to the main purpose.

What counts as proof?

The rule in my instruction files: don't say something is fixed, done, live, working or verified without evidence appropriate to the task. In practice that means:

• Screenshots of the agent using the feature like a person would
• The live link, opened and checked, not the local preview
• Rendered output for anything visual or written
• Test results where tests exist
• A real end-to-end run, even if it costs a little

That last one matters. I tell my agents not to worry about 10, 20 or 30 cents, even $1 of usage. Just prove it works. I'm happy to spend 20 cents to know it truly works. I can't be a bug fixer, and I don't want to open the builder and find another obvious dumb mistake.

The person who built it does a lightweight self-check before showing me anything meaningful. Independent review is based on risk. And I shouldn't be the primary tester of my own software.

When is a deploy not done?

When the thing it was supposed to do hasn't happened yet. On one scheduled job, the same bug survived three rounds of merge, deploy, discover the bug. Before I'd accept the fourth, the instruction was:

Trigger the job manually after the deploy.
Wait for it to finish (not just deploy).
Confirm a real row lands in the log.
Only then run the final tests and the report.
If it hangs or errors again, stop and report.

Deployed is a step. A real result landing where it should is the finish.

I set up a deployment confirmation step, too, so every change tells me whether it's live or still local. Ask your agent to end every task by saying which one it is, with the link.

Why does the same bug keep coming back?

Because agents fix symptoms. They reach for the quickest, simplest little fix. On my survey software, my coding agent found it had made about 14 earlier "fixes" over weeks, each one tuning a single query. The real cause was an undersized database server. A $5 a month upgrade took errors from 229 per 30 minutes to zero, according to the agent's own report.

When a fix fails more than once, I tell the agent to step back and look at it from first principles. And I don't want to hear from it until it can do three things:

1. Explain in detail what the actual problem was
2. Explain why it missed it before
3. Confirm it tested the result itself, end to end, and that it worked

The short check I said yes to, before any fix:

Before you fix anything:
1. Name the actual error.
2. Check whether we're hitting a platform limit.
3. Check whether something already solves this.

What if it built a lot and still missed the point?

It happens. I spent about 30 hours over a month building a new version of an AI agent system for Slack. The tool built insane stuff, docs and features everywhere. It still didn't do the one thing the system exists for: an agent in your Slack that acts like a human you can chat with and that does things for you.

I had a different AI analyze it. Its verdict: not broken, not trash, just five or six things to fix. I think it may have solved about 90% of the issue in about four hours. The lesson for me: state the main purpose plainly, and have a second AI audit the work against that purpose.

What if the agent is confidently wrong?

Push back. AI is smart, but sometimes it misreads the goal. I asked an agent for a working folder for my Local Media HQ business, and it told me to start a whole second project so I wouldn't touch the one with the website. That's dumb. Every other project I have keeps the website and the business in one place. Why double my surface area? When I said so, it agreed right away.

I caught it because I'd seen that failure before. Know the patterns and you can say, no, you're just flat out wrong.

Should the agent check its own documents, too?

Yes. Why would you not read a document you had created? An agent should re-read and check its own work before handing it over. Read it as a first-time user, not as the writer: does this sentence make sense to someone who doesn't know what's going on? For bigger outputs, I want a separate reviewer agent in the loop: should an AI reviewer check work before you see it.