Generating Tests With AI That You'd Actually Keep in the Repo

Ask an AI to “write tests for this function” and you will get tests. You will get a lot of them, they will be green, and roughly half will be worthless. The failure mode is specific and predictable: the model reads your implementation, sees that it returns 7 for a given input, and writes assert result == 7. That test doesn't verify the function is correct. It verifies the function hasn't changed. Those are wildly different things, and a suite full of the latter is a tax, not an asset — it breaks on every legitimate refactor and catches nothing.

Good AI-generated tests are absolutely achievable. You just have to stop the model from reading the answer key.

Test against the spec, not the implementation

The core trick is to give the model the contract and withhold the body, or at least explicitly forbid it from asserting on observed output. When the model has to derive the expected value from the requirement rather than copy it from the code, the tests become meaningful.

Write tests for a function `prorate(amount, start, end, period)`
that returns the fraction of `amount` owed for a partial billing
period. Contract:
- a full period returns amount unchanged
- half a period returns amount/2
- start == end returns 0
- end before start is an error

Derive every expected value from this contract by hand. Do NOT
run or read the implementation to find expected outputs. Show
your arithmetic in a comment above each assertion.

The “show your arithmetic” instruction is the tell for whether it obeyed. If the comment says // 30 days of 30 = full period -> 500, the model reasoned from the spec. If the assertions are suspiciously precise floats with no derivation, it peeked, and you throw them out.

The property test is where AI earns its keep

Example-based tests are easy to write and easy to write badly. The place AI genuinely accelerates good testing is property-based tests, because the hard part there is inventing the invariant, and models are good at enumerating invariants once you prime them.

For this JSON serializer, list every property that must hold
for all inputs, then write Hypothesis tests for each. Start
with the obvious one (round-trip: parse(dump(x)) == x) and
then find at least four non-obvious ones.

The round-trip property alone is worth more than twenty hand-picked examples, because it runs against thousands of generated inputs and finds the unicode edge case you never thought of. The model will also usually surface things like “dump output is valid JSON,” “dumping is deterministic,” and “key order doesn't affect equality” — each a real property that a human writing examples tends to skip.

A concrete result: on a decimal-money library, an AI-suggested property that “addition is associative across generated amounts” found a rounding bug that had survived two years of example tests, because the failing triple was three specific values no human would pick.

Make it write the test that fails first

If you're fixing a bug, the highest-value AI test is the regression test, and the discipline is the same as human TDD: the test must fail on the current code. Models will cheerfully write a “regression test” that passes against the buggy version, which proves nothing.

This is the bug report: negative quantities are accepted and
create negative inventory. Write a test that reproduces it.
The test MUST fail against the current code. I will run it
before fixing anything. If it passes now, you've written the
wrong test.

Then actually run it before the fix. Watching it go red is the only proof that it pins the bug and not a coincidence. Cursor and Claude Code make this cheap because they can run the test themselves and iterate until it fails for the stated reason — but you still read the failure message to confirm it's failing on the assertion, not on an import error.

Coverage is a trap the model will walk you into

Ask for “100% coverage” and you'll get it, along with tests that call every branch and assert almost nothing meaningful. Coverage measures which lines ran, not which behaviors are pinned. A test that calls divide(6, 2) and asserts nothing about the result covers the line perfectly.

Instead of coverage targets, ask for the behaviors:

Don't optimize for line coverage. List the distinct BEHAVIORS
of this module a user depends on, including error behaviors and
boundary behaviors. Write one focused test per behavior with a
name that states the behavior. If two tests would assert the
same behavior, keep one.

The test names alone become documentation: test_refund_exceeding_balance_is_rejected tells the next reader what the system guarantees. test_refund_case_3 tells them nothing.

What to review before you commit AI tests

Read every generated test with three questions:

• Would this fail if the behavior broke? Mentally introduce the bug — flip a comparison, drop a validation — and check the test would go red. If you can't imagine a break that fails it, delete it.
• Is the expected value derived or copied? Copied-from-output assertions are change detectors, not correctness checks. Keep them only where you genuinely want to pin exact output (snapshot of a serialized format, say) and label them as such.
• Is it testing my code or the language? Models love to test that the standard library works. A test that a Python list append adds an element is pure noise.

The mocking failure mode

Given a function with dependencies, AI reaches for mocks aggressively, and over-mocked tests are the ones most likely to pass while the system is broken. A test that mocks the database, mocks the clock, mocks the network, and then asserts the mocks were called has tested that your code calls functions — not that anything works. When the model mocks something, ask whether the real thing is cheap enough to use instead. An in-memory SQLite or a real temp file usually beats a mock and catches the integration bugs mocks paper over.

The workflow that sticks

What has held up on my team: AI generates the property tests and the edge-case enumeration (its genuine strength), a human writes or closely reviews the handful of core behavioral tests, and every regression test is proven to fail before its fix lands. We let the model draft, we run everything, we delete a third of what it produces without guilt, and we never let a “test” into the repo that we haven't watched fail for the right reason at least once. The suite that results is smaller than what the AI first offered and far more useful than what we'd have written by hand under deadline.