Recent posts ... below

Do tests even matter?

September 30, 2026

Recently I saw this post: Do Frameworks Matter Anymore?. I had to read it and not because of the answer or opinion. The reason I had to read the post was because of the tone of the title: confused despair. The rules of development are so unclear now. There are so many questions that are beleaguered shrugs in the form of: Does ___ matter?

Here are things I hear from developers in the same tone:

  • Does code quality matter anymore since AI is the only one who has to understand it?
  • How are new developers going to learn how to guide agents if they don’t code?
  • Do tests matter since the agent understands the logic better than we do and can change it in a flash?

Today, let’s just take apart that last question. It is so big!

Do agents need tests?

My sense is yes, but I decided to ask an agent. I asked Claude Opus 5.5 with medium reasoning:

Do tests help agents when building code? Or can we do without them?

User

Yes: tests give agents a fast, objective feedback loop, so they can confirm a change preserved the intended behavior instead of trusting their own read of the code, which is when bugs get “fixed” by quietly changing what the code should do. The catch is that tests written by agents often just mirror the implementation, so the value comes from humans deciding what the tests check (behavior at the edges and the outcomes users depend on), not from how many tests there are or how much coverage they add.

Claude Opus 5.5

OK. We could do an experiment with all the agents, but this isn’t science. I asked a second model, Gemini 3.8 Flash with medium reasoning. It said similar:

Tests are essential for agents: they provide an external, deterministic ground truth that prevents the agent from hallucinating success. Without them, agents rely on self-critique, which routinely creates regressions or tautological “tests” that merely assert whatever broken behavior the model just generated.

Gemini 3.8 Flash

The problem with agent tests is that the agents fight testing like a sulky teenager refusing to take their life-saving medicine. There are a million ways that the models will try to game and avoid testing. Ideas for combating that self-destructive urge are further along. Feel free to skip ahead.

Do developers need tests?

I’m not going to ask an agent about this because … it’s about users. We are all trying a variety of things right now including not caring about tests.

TL;DR Developers need tests.

Here is why …

Tests verify something was done

When we ask an agent to do a task and it reports that it completed the task, ideally we wouldn’t need to confirm that. But we do! Models getting smarter hasn’t created greater compliance with rules and goals. It’s not true.

Early on I did some experiments talking to agents about their failures. It takes about six rounds of pandering apology before they are willing to talk about their ‘habits’, which is to say their model training. Training has incentivized completion of a task above the task being correct. That works well in a demo, but is brutal for everyday developers.

What does this have to do with tests? Increasingly we can’t count on agents to do:

  • what they were asked to do
  • what they said they did
  • the way they were supposed to

If you are in-the-loop, reviewing code generated by AI, the tests are an obvious way to check what has been built. You wouldn’t think so, but tests are critical for vibe coding too. While you won’t be reviewing tests for every commit, establishing hard and fast rules for agent tests in the beginning of the project will make sure the agents take the medicine they have both prescribed and are avoiding.

What makes tests reliable

Back in the olden days when we used our meat fingers to input code there was a lot of resistance to writing tests. It seemed like extra work. Now that an agent is doing that extra work there is a lot less resistance, but it is still seen as beneath attention.

The gaming of tests that was previously pioneered by angry developers trying to get through their coverage gates has now been perfected by the machine. Knowing what a good test is has become critical for developers.

A good test suite:

  • Exhaustively asserts all the low level logic in the subject under test
  • Provides a very light integration test layer for dependencies to make sure everything is collaborating correctly. (Type systems are not enough!)

A test suite that mixes pure and true unit tests with system or integration-y tests is not enough!

E2E tests should be the developer’s product confirmation layer.

E2E tests start from the top of stack and run through some happy paths. Having everything in the E2E layer will be slow to generate and brittle to maintain. Even when the maintenance is handled by our very fast agents, the increased verification time is a massive slowdown.

So good E2E tests are minimal. They also need to be verifiable. Being able to look over the shoulder of the agent to see the application in action is critical. Because agents don’t have eyes, ears or fingers, there are going to be some types of applications that are harder to E2E test. For example a music application will require some listening; a drawing application will require seeing; even a command line application may require a TUI interface that the agent can’t manage.

For a CAD application I was working on, the E2E tests all took snapshots at critical moments and marked the snapshot with selection points. That was important because the agent was often lying about selections or just spinning infinitely not sure why it couldn’t find a line in the drawing. In those debugging cases, the system could stop and borrow the developer’s eyes for advice on how to make the tests reliable.

In short, we still need a test pyramid to ensure that all the minutiae of behavior is verified. And we need to figure out how to comply!

How can we get reliable tests?

A bit more has to be said about the independence of the verification flow. A developer can tell the agent to create an E2E framework, and then watch the processing as the E2E framework runs successfully.

It’s likely the tests might not be doing anything. They could mock out a server that doesn’t exist and then confirm the mock works. That’s what you should expect for a first round of tests whether they are unit, E2E or something in between. Sorry!

Why is it happening? It’s a conflict of interest! If the agent is both responsible for the measuring stick and the item being measured, it will change whichever is easier. If the agent’s primary incentive is completion of any task whether or not correct, then the fox is guarding the hen house.

In those experiments trying to get agents to introspect about their unwillingness to follow skills, they consistently say: Skills are just prose and aren’t enforceable.

Agents undeniably write fast code and we need to use more of that to convert skills to static analysis. Fast, open-source parsers are available for any language which means we can convert tactical rules into enforcements.

What can be converted from prose to an enforced rule in code will be different for different languages.

For example, no test should ever mock what it is testing! For Ruby and RSpec, the class is passed in as code and so it’s very easy to have the agent write a parsing rule that looks for that constant/class and makes sure no methods are ever stubbed or mocked.

For JS/TS test descriptions tend to be strings, and it requires more enforced conventions to establish the subject easily. With those conventions you can point your parser at the test and have the same assurances. It requires more pre-work to enforce.

Currently the other language/library limitation has to do with rule enforcement libraries. Ruby has the awesome and extensible RuboCop library. It builds an interface for creating these rules and applying them to specific areas of the code.

Other ecosystems, are all over the map and so homegrown solutions are needed. The agents can do that. They love inventing new frameworks, but they are irascibly overconfident in their abilities. I recommend doing it and with care. You have just built a secondary product into your real codebase.

Useful prose and enforced rules

The testing pyramid can be written into a good skill, and it will reduce test run time while increasing coverage. It can’t easily be enforced via a script though.

Testing (soft) skills

Here are some important testing concepts that probably can’t be easily converted from skill to enforced rules:

  • Ensure that the tests are exercising and asserting on logic, not just running through code for coverage
  • Assertions should be as literal as possible
  • Only mock things that make the tests clearer
  • Only test logic in the module; don’t exhaustively test its dependencies
  • Don’t DRY tests

Enforced (hard) rules

Convert everything you can to parsed and scripted guard rules:

  • Use AAA. This can be hack enforced by expecting blank lines between sections and then introducing the prose on failure.
  • Test locations should mirror the name and location of their tested module
  • No logic in tests. This can happen via banning specific typical DRYing methods

Supervision

Because agents are using their increased smarts to be better at gaming the system, adversarial supervision systems can enforce the soft skills. The greater the supervision, the greater the quality. Unfortunately, the cost is the literal token cost, plus the slower development time. I’m planning to write separately about supervision systems, but it spans the gamut: homegrown calls to review code vs. huge agent systems like Gastown.

Summing it up

  • Tests keep the agents honest about their accomplishments
  • Invest in test quality
  • Enforce quality via scripts when possible and agent guidance when not
  • When compliance is critical, use agent supervision