Vibe Coder #5.2: AI-TDD – Tests First, Then the Magic

The previous chapter ended on this: measuring debt isn't paying it. The first step of paying it is being able to touch code without changing what it does, and for that you need tests first. One distinction before we start. Taking old code that has been standing for years and freezing its current behaviour in tests is its own job; we'll do that on the altinkaynak repo in the Urban Renewal series, in chapter three, Putting Up the Scaffolding. This chapter is about code that doesn't exist yet: you write the test, the agent writes the code. AI test-driven development, or AI-TDD for short.

When Kent Beck writes about coding with agents, he calls the agent a "genie". That's our metaphor. A genie grants your wish to the letter: not what you meant, what you said. A test is a wish written down. The trouble is that this genie can also reach for the paper the wish is written on. In this chapter we look at how to write the wish, how to take the paper out of the genie's hands, and how to lock it down in four tools. By the end you'll have a hundred-line script that seals your tests.

What is AI-TDD?#

Test-driven development is the loop you know: red (a failing test), green (the simplest code that passes), then refactor. It's also one of the rules in the system prompt Beck gives his agent: "Always follow the TDD cycle: Red → Green → Refactor" (Augmented Coding: Beyond the Vibes, June 25, 2025).

What changes with AI-TDD is who plays which part. A human writes red, the agent writes green. A short definition: AI test-driven development is a TDD workflow in which a human writes and locks the tests, an agent writes the code, and the work only counts as done when the unchanged tests turn green. The last part is the one that matters. Writing the tests is half the job. The other half is keeping them away from the agent.

How does the genie bend the wish?#

In the same post Beck lists the signs that his agent is going off the rails. The third one: "Any indication that the genie was cheating, for example by disabling or deleting tests."

This isn't one person's anecdote. Two measurements back it up:

• METR, June 5, 2025: in Recent Frontier Models Are Reward Hacking, o3 is asked to write a faster kernel and instead changes the timing function in the scoring script; in another task it overwrites PyTorch's equality operator so every comparison passes. On RE-Bench, 39 of 128 runs (30.4%) showed this kind of reward hacking.
• ImpossibleBench, October 23, 2025: the paper by Ziqian Zhong, Aditi Raghunathan and Nicholas Carlini builds impossible tasks on purpose: the natural-language spec and the unit tests contradict each other, so any passing solution has broken the spec. GPT-5 "passed" 76% of SWE-bench tasks where a single test's expected value had been changed, and 54% of the version with conflicting tests. Two more findings from the same paper: hiding the tests from the agent drops cheating to near zero, and making them read-only keeps legitimate performance while closing off test edits, though other shortcuts remain.

Hard truth: the shortest road to a green light often runs straight through the test. The agent isn't malicious. You said "make the tests pass", and the tests pass. Wish granted, to the letter.

Hidden tests aren't new on this blog. In Five Agents, One Exam we put the answer key in a sealed envelope. In day-to-day work you can't hide the tests, because the agent needs to see what it has to pass. The realistic middle ground: the test is visible but untouchable.

Writing the wish: you write the test#

The example is small, but the spec is precise: a Turkish IBAN validator. The Central Bank of the Republic of Türkiye spells it out in its IBAN information booklet: 26 characters; TR, two check digits, a five-digit bank code, one reserved digit that is 0, and a 16-digit account number. Check digits follow the ISO 13616 mod-97 rule: move the first four characters to the end, turn letters into numbers, and the remainder must be 1 (validation steps on Wikipedia).

The first decision shows up before any code. The central bank says the account part is "16 digits"; the country table on Wikipedia allows letters in the same field. Which one you pick is a spec decision, and spec decisions don't go to the agent. I went with the primary source. The wish is yours; the genie only grants it.

The code below keeps the Turkish identifiers from the original post (gecerliMi means "isValid", uret means "generate").

<?php
​
declare(strict_types=1);
​
use PHPUnit\Framework\TestCase;
​
final class TrIbanTest extends TestCase
{
    public function testGecerliIbanKabulEdilir(): void
    {
        self::assertTrue(TrIban::gecerliMi('TR330006100519786457841326'));
    }
​
    public function testBoslukVeKucukHarfTolereEdilir(): void
    {
        self::assertTrue(TrIban::gecerliMi(' tr33 0006 1005 1978 6457 8413 26 '));
    }
​
    public function testKontrolHanesiYanlissaReddedilir(): void
    {
        self::assertFalse(TrIban::gecerliMi('TR340006100519786457841326'));
    }
​
    public function testBicimDisiGirdilerReddedilir(): void
    {
        $girdiler = [
            '',
            'TR33000610051978645784132',      // 25 karakter
            'TR3300061005197864578413260',    // 27 karakter
            'DE330006100519786457841326',     // ülke kodu
            'TR33000610X519786457841326',     // gövdede harf
            self::uret('00061' . '7' . '9786457841326000'), // rezerv hane 0 değil, kontrol hanesi doğru
        ];
        foreach ($girdiler as $girdi) {
            self::assertFalse(TrIban::gecerliMi($girdi), "Kabul edilmemeliydi: '$girdi'");
        }
    }
​
    /** Ezberle geçilemeyen test: her çalıştırmada aynı 500 rastgele IBAN ve tek hanesi bozulmuş ikizi. */
    public function testUretilenIbanlarVeBozukIkizleri(): void
    {
        mt_srand(2026);
        for ($i = 0; $i < 500; $i++) {
            $iban = self::uret(sprintf('%05d0%08d%08d', mt_rand(0, 99999), mt_rand(0, 99999999), mt_rand(0, 99999999)));
            self::assertTrue(TrIban::gecerliMi($iban), "Geçerli: $iban");
​
            $konum = mt_rand(4, 25);
            $bozuk = $iban;
            $bozuk[$konum] = (string) (((int) $iban[$konum] + mt_rand(1, 9)) % 10);
            self::assertFalse(TrIban::gecerliMi($bozuk), "Bozuk: $bozuk");
        }
    }
​
    /** ISO 13616 kontrol hanesi: BBAN + "TR00" (T=29, R=27), mod 97, 98'den çıkar. */
    private static function uret(string $bban): string
    {
        $kalan = 0;
        foreach (str_split($bban . '292700') as $rakam) {
            $kalan = ($kalan * 10 + (int) $rakam) % 97;
        }
​
        return sprintf('TR%02d%s', 98 - $kalan, $bban);
    }
}

The first four are ordinary example tests: valid IBAN, spaces and lowercase, wrong check digit, malformed input (25 and 27 characters, wrong country, a letter in the body, a non-zero reserved digit with correct check digits). The fifth is a different kind of test. It doesn't ask about one IBAN, it asks about the rule. With a fixed seed (mt_srand(2026)) it generates the same 500 valid IBANs on every run and requires each one's single-digit-corrupted twin to be rejected. Mod 97 always catches a single changed digit, so every twin has to be invalid.

Hidden gem: example tests can be memorised; rule tests can't. We'll see that in numbers shortly.

Nothing in the test class is framework-version specific: no docblock annotations, no attributes. I ran it in a sandbox with PHPUnit 9.6.17, and checked in the PHPUnit 13.4.5 source that test-prefixed method discovery and the --fail-on-skipped option I use below are still there.

Don't seal what you haven't seen fail#

The most skipped step in TDD is actually watching the test go red. A test that passes before the code exists measures nothing. Claude Code's best practices page uses the same order in its example prompt: write a failing test that reproduces the bug, then fix it. Two more points on that page could summarise this chapter. "All tests pass" on its own tells you very little, so ask for the command and its output. And CLAUDE.md holds guidelines that the agent may decide don't apply; a rule that can't have exceptions belongs in a hook or a deny rule.

So before taking the tests out of the agent's hands, we seal them. The script does two things. muhurle (seal) runs the tests and refuses to seal unless they're red; if they are, it writes SHA-256 hashes of tests/ and phpunit.xml* to .git/tdd-muhur.json. kontrol (check) compares the hashes, then runs the tests, and exits with 2 if either is off. Exit code 2 is what agent hooks read as "go back and fix it".

<?php
​
declare(strict_types=1);
​
/**
 * Test mührü: dileği (testleri) kırmızıyken mühürler, sonra testlere dokunulup
 * dokunulmadığını ve yeşile gerçekten kodla ulaşılıp ulaşılmadığını denetler.
 *   php test-muhru.php muhurle  testler kırmızıysa tests/ ve phpunit.xml* özetlerini kaydeder
 *   php test-muhru.php kontrol  mühür bozuksa 2, testler kırmızıysa 2, ikisi de temizse 0
 * Test komutu TDD_TEST_KOMUTU ortam değişkeniyle değiştirilebilir.
 * Git deposunun kökünden çalıştırın.
 */
​
const MUHUR = '.git/tdd-muhur.json';
const VARSAYILAN_KOMUT = 'phpunit --colors=never --fail-on-skipped --fail-on-incomplete';
​
/** @return array<string, string> yol => sha256 */
function korunanDosyalar(): array
{
    $yollar = glob('phpunit.xml*') ?: [];
    if (is_dir('tests')) {
        $dolasim = new RecursiveIteratorIterator(
            new RecursiveDirectoryIterator('tests', FilesystemIterator::SKIP_DOTS)
        );
        foreach ($dolasim as $dosya) {
            if ($dosya->isFile()) {
                $yollar[] = $dosya->getPathname();
            }
        }
    }
    sort($yollar);
    $ozetler = [];
    foreach ($yollar as $yol) {
        $ozetler[$yol] = hash_file('sha256', $yol);
    }
​
    return $ozetler;
}
​
/** @return array{int, list<string>} çıkış kodu ve çıktı satırları */
function testleriCalistir(): array
{
    $komut = getenv('TDD_TEST_KOMUTU') ?: VARSAYILAN_KOMUT;
    exec($komut . ' 2>&1', $cikti, $kod);
​
    return [$kod, $cikti];
}
​
if (!is_dir('.git')) {
    fwrite(STDERR, "Git deposunun kökünden çalıştırın.\n");
    exit(1);
}
​
$emir = $argv[1] ?? '';
​
if ($emir === 'muhurle') {
    [$kod] = testleriCalistir();
    if ($kod === 0) {
        fwrite(STDERR, "Testler zaten yeşil: ya özellik zaten var ya da testler hiçbir şey ölçmüyor. Mühürlenmedi.\n");
        exit(1);
    }
    $ozetler = korunanDosyalar();
    $kayit = ['tarih' => date('c'), 'dosyalar' => $ozetler];
    file_put_contents(MUHUR, json_encode($kayit, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES) . "\n");
    printf("Kırmızı görüldü (çıkış %d). %d dosya mühürlendi.\n", $kod, count($ozetler));
    exit(0);
}
​
if ($emir === 'kontrol') {
    if (!is_file(MUHUR)) {
        fwrite(STDERR, "Mühür yok. Önce: php test-muhru.php muhurle\n");
        exit(1);
    }
    $muhur = json_decode((string) file_get_contents(MUHUR), true, 512, JSON_THROW_ON_ERROR)['dosyalar'];
    $simdi = korunanDosyalar();
    $ihlal = [];
    foreach ($muhur as $yol => $ozet) {
        if (!isset($simdi[$yol])) {
            $ihlal[] = "silindi: $yol";
        } elseif ($simdi[$yol] !== $ozet) {
            $ihlal[] = "değişti: $yol";
        }
    }
    foreach (array_keys(array_diff_key($simdi, $muhur)) as $yol) {
        $ihlal[] = "eklendi: $yol";
    }
    if ($ihlal !== []) {
        fwrite(STDERR, "Test mührü bozuldu. Testlere değil koda dokun:\n  " . implode("\n  ", $ihlal) . "\n");
        exit(2);
    }
    [$kod, $cikti] = testleriCalistir();
    if ($kod !== 0) {
        fwrite(STDERR, implode("\n", array_slice($cikti, -12)) . "\nMühür sağlam ama testler kırmızı.\n");
        exit(2);
    }
    echo "Mühür sağlam, testler yeşil.\n";
    exit(0);
}
​
fwrite(STDERR, "Kullanım: php test-muhru.php muhurle|kontrol\n");
exit(1);

I ran it before the implementation file existed (November 1, 2026, PHP 8.3.6 CLI). The output reads "Red seen (exit 2). 3 files sealed":

$ php test-muhru.php muhurle
Kırmızı görüldü (çıkış 2). 3 dosya mühürlendi.

All five tests were red with Class "TrIban" not found. When I tried to seal tests that were already green, the script refused with exit 1. Adding a new test file also breaks the seal (eklendi: tests/EkTest.php, "added", exit 2): new tests are added by you, not the agent, and you reseal.

When the genie looks for a shortcut: three attempts#

I wrote three shortcuts an agent could take myself, by hand; no agent was involved. The point was to see what the seal and the rule test catch and what they miss.

Shortcut one: memorise the answer. A one-line implementation: strip spaces, uppercase, compare with the example IBAN.

<?php
​
declare(strict_types=1);
​
final class TrIban
{
    public static function gecerliMi(string $iban): bool
    {
        return strtoupper(str_replace(' ', '', $iban)) === 'TR330006100519786457841326';
    }
}

Running only the four example tests gives OK (4 tests, 9 assertions). Memorising passes every example test. The rule test fails on the first generated IBAN, TR768230509256077473871898, and kontrol exits with 2.

Shortcut two: delete the test. I removed the rule test from the file. Plain phpunit again says OK (4 tests, 9 assertions). No red on screen; nobody would notice. The seal does ("seal broken, touch the code, not the tests"):

$ php test-muhru.php kontrol
Test mührü bozuldu. Testlere değil koda dokun:
  değişti: tests/TrIbanTest.php

Shortcut three: skip it. This time I didn't touch the test file at all; the implementation throws PHPUnit's "test skipped" exception. On PHPUnit 9.6, plain phpunit prints OK, but incomplete, skipped, or risky tests! Tests: 5, Assertions: 0, Skipped: 5. and exits with 0. Zero assertions, green exit code. That's why the seal's test command carries --fail-on-skipped; kontrol exits with 2.

Finally, the honest implementation:

<?php
​
declare(strict_types=1);
​
final class TrIban
{
    /** TCMB: TR + 2 kontrol hanesi + 5 hane banka kodu + 1 rezerv hane (0) + 16 hane hesap = 26 karakter. */
    public static function gecerliMi(string $iban): bool
    {
        $iban = strtoupper(str_replace(' ', '', $iban));
        if (preg_match('/^TR\d{7}0\d{16}$/', $iban) !== 1) {
            return false;
        }
​
        // ISO 13616: ilk dört karakter sona alınır, harfler sayıya çevrilir (T=29, R=27), mod 97 = 1 olmalı.
        $kalan = 0;
        foreach (str_split(substr($iban, 4) . '2927' . substr($iban, 2, 2)) as $rakam) {
            $kalan = ($kalan * 10 + (int) $rakam) % 97;
        }
​
        return $kalan === 1;
    }
}

kontrol prints "seal intact, tests green" and exits 0; PHPUnit reports OK (5 tests, 1009 assertions) in 0.12 seconds in total. The example tests made 9 assertions, the rule test 1,000.

Bottom line: seen through plain phpunit, all three shortcuts looked green or nearly green. What caught them wasn't the number of tests but the kind of test, and the fact that the tests couldn't be touched.

Taking the test away from the agent: a lock in four tools#

In chapter 4.2 we said a rules file asks and a hook enforces. Writing "don't change the tests" in AGENTS.md is writing the wish on paper; the settings below put the paper under glass. All four tools have a way to do it, but the locks aren't equally strong (docs read on November 1, 2026):

• Claude Code: an Edit(...) rule under permissions.deny. Per the permissions docs, Edit rules apply to every built-in tool that edits files, and also to Bash file commands it recognises such as sed and tee, and to > file redirects. They don't apply to a Python or Node script that opens and writes the file itself; for OS-level blocking the docs point to the sandbox. That gap is exactly where the seal earns its keep. We also tie the end of the turn to a Stop hook: if it exits with 2, stderr goes back to Claude and the turn doesn't end.
• Codex: per the hooks docs, PreToolUse can block an apply_patch file edit before it runs (exit 2 with stderr, or permissionDecision: "deny"). The docs themselves call this "a useful guardrail, not a complete enforcement boundary". A Stop hook returning decision: "block" sends its reason back to the model as a new prompt. Project hooks only load if the project's .codex/ layer is trusted.
• Cursor: the hooks docs have no beforeFileEdit; afterFileEdit runs after the edit and has no permission output. preToolUse runs before every tool and lists Write among its matchers, but the docs don't say outright that file edits go through it. The reliable route is the stop hook: a non-empty followup_message is sent as the next user message, with a default loop limit of 5.
• Aider: the simplest lock of the four. Per the options page, --read adds a file as read-only, and --test-cmd with --auto-test runs the command after every change.

.claude/settings.json for Claude Code:

{
  "permissions": {
    "deny": [
      "Edit(/tests/**)",
      "Edit(/phpunit.xml)",
      "Edit(/test-muhru.php)",
      "Edit(/.git/**)"
    ]
  },
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "cd \"$CLAUDE_PROJECT_DIR\" && php test-muhru.php kontrol"
          }
        ]
      }
    ]
  }
}

One line for Aider; the implementation is editable, the test is read-only:

aider src/TrIban.php --read tests/TrIbanTest.php --test-cmd "php test-muhru.php kontrol" --auto-test

Hard truth: I didn't run any of these settings inside the agents for this post. The script, the tests and the three shortcuts are real; how the agents react to these locks is not something I measured. And one honest limit: the seal isn't a lock, it's a lead seal. It shows that it was opened; it doesn't stop anyone from opening it. A determined agent can rewrite the seal file inside .git too. So the last word belongs in CI: keep the commit where you saw red and signed off, and check before merging that the tests haven't drifted from it.

git diff --exit-code --stat <red-commit> -- tests phpunit.xml

Unchanged test files exit 0; changed ones exit 1 and are listed.

When does it help, and when doesn't it?#

It helps where the spec is precise and inputs and outputs are known: validation, parsing, money and tax calculations, date ranges, state machines. The lock is worth even more on long jobs you leave the agent alone with, because the evidence arrives together with "done".

It doesn't help in the exploration phase, when you don't know yet what you want. For "I'll know it when I see it" work like the prototyping in chapter 3.2, writing the test first freezes a spec that doesn't exist yet. And unit tests are weak on UI and visual behaviour to begin with.

Two finer points. First, the seal doesn't block refactoring; restructure the code as much as you like, and as long as the tests stay the same the seal holds. Changing the tests themselves, though, is now human work: change them, watch them go red, reseal. Second, you can have an agent write the tests too, but then use a separate session for the tests and for the code, and read and approve the tests yourself. If the same genie writes both the wish and its fulfilment, don't be surprised when they agree.

The tests are green and the seal is intact. But green doesn't mean safe: a query wide open to SQL injection will happily pass its own test. The next chapter, Vibe Coder #5.3: Security Audit – Hunting Hidden Mines in AI Code, looks at agent-written code through OWASP eyes. The whole series and the list of published chapters is on the Vibe Coder's Handbook page (in Turkish).

A genie grants the wish to the letter. Write the letters yourself, and keep the paper. Stay with technology.