Urban Renewal #3: Scaffolding – Freeze the Behaviour, Then Touch It

Urban Renewal #2 ended with a decision: the Altınkaynak library doesn't get torn down, it gets retrofitted. The SOAP plumbing comes out, a layer that reads JSON goes in, and the public methods stay where they are. But before any plumbing comes out there's one more job: record how the building stands today, crooked walls included. This chapter is about characterization tests: tests that describe what the code does today, not what it should do.

On a renovation site the first thing to go up is the scaffolding. Scaffolding doesn't fix the building and it doesn't hold it straight. It catches whatever falls while the crew is working. A characterization test is the same thing. It doesn't make the code correct; it tells you when something moves.

For this post I bolted 18 tests onto altinkaynak without changing a single line of its source, then deliberately broke the code seven ways to see whether the scaffolding actually bears load. My first scaffold missed two of the seven.

A note on names: the repository's tests are in Turkish, and the code below is quoted exactly as it sits in the branch. Sahte means fake, iskele means scaffolding, BilinenHata means known bug.

Write down what is, not what should be#

The term comes from Michael Feathers, who used it in Working Effectively with Legacy Code and explained it again in a short 2016 post, Characterization Testing. The gist: these tests exist to document what the system actually does, not to check for the behaviour you wish it had. The method is simple. Write a test with a deliberately wrong expectation, run it, read the real output in the failure message, and make that output the expected value. He adds one more piece of advice: if you find a bug along the way, it's often a good idea to leave the test in place.

We covered TDD in Vibe Coder 5.2. Both use the same tool, PHPUnit, but they run in opposite directions. In TDD the test comes before the code and states the behaviour you want. With a characterization test the code is already there; the test follows it and records what it does. In one a red test means "you haven't written it yet", in the other it means "something changed".

Hard truth: the hardest part of writing characterization tests is not fixing the bug right in front of you. That goes for the agent and for us.

The seam: where the building meets the grid#

First, find where the building talks to the outside world. The Altınkaynak class builds its SOAP client on the first call and keeps it in a private $Service property. If the property is already set, it doesn't build a new one. In Feathers' terms that's a seam: a place where you can change the program's behaviour without editing the code at that spot. Martin Fowler's Legacy Seam quotes the definition and describes seams as a way to break dependencies for testing.

We drop a fake service that never touches the network into that property:

<?php
​
declare(strict_types=1);
​
namespace Altinkaynak\Tests;
​
use Altinkaynak\Altinkaynak;
use ReflectionProperty;
​
final class SahteServis
{
    /** @var list<array{metot: string, argumanlar: array<mixed>}> */
    public array $cagrilar = [];
​
    /** @param array<string, string|\Throwable> $yanitlar metot adı => XML metni ya da fırlatılacak hata */
    public function __construct(private array $yanitlar)
    {
    }
​
    /** @param array<mixed> $argumanlar */
    public function __soapCall(string $metot, array $argumanlar): object
    {
        $this->cagrilar[] = ['metot' => $metot, 'argumanlar' => $argumanlar];
        $yanit = $this->yanitlar[$metot] ?? throw new \LogicException("Beklenmeyen çağrı: {$metot}");
​
        if ($yanit instanceof \Throwable) {
            throw $yanit;
        }
​
        return (object) [$metot . 'Result' => $yanit];
    }
​
    /** Kaynağa dokunmadan sahteyi özel $Service özelliğine koyar. */
    public static function tak(Altinkaynak $istemci, self $sahte): void
    {
        (new ReflectionProperty(Altinkaynak::class, 'Service'))->setValue($istemci, $sahte);
    }
}

The fake answers __soapCall() with canned XML (or throws a canned error) and logs every call; tak() ("attach") puts it in place through reflection. This seam has a cost, and I'll say it up front: it depends on the name of a private property. Rename the property and the scaffolding collapses. It will be the first piece of scaffolding to come down, because the retrofit removes the SOAP layer anyway.

A test caught the second detail. setUsername(), setPassword() and setServiceURL() all reset $Service to null. Change a setting inside a test after attaching the fake, and the next call builds a real SOAP client and tries to go out on the network. You've taken the scaffolding down with your own hands.

The approved copy: output from 2020#

Where does the expected value come from? Altinkaynak has one advantage: the 2020 post (in Turkish) contains a sample loop over GetMain() and its printed output. That output is the oldest, most real record of this code's behaviour we have. I copied it character for character into a file and made it the test's "approved" copy. The technique also goes by golden master or approval testing.

One thing I need to be upfront about. The SOAP service no longer answers, so I couldn't record a real 2020 SOAP response. The XML file that feeds the test is a reconstruction: field names from what the code reads, values from the 2020 output. I don't know the real name of the root element; SimpleXML never reads it, so it doesn't affect the result. A comment at the top of the file says all of this.

The test runs the exact loop from the 2020 post and compares its output with the approved file. The three public methods are three copies of the same parsing code, so all three are checked separately (why that matters comes further down):

/** @dataProvider metotlar */
public function test2020YazisindakiCiktiyiAynenUretir(string $metot): void
{
    $istemci = $this->istemci([$metot => (string) file_get_contents(self::FIXTURE . 'getmain-2020.xml')]);
​
    $this->assertSame(
        (string) file_get_contents(self::FIXTURE . 'getmain-2020.approved.txt'),
        $this->yazdir($istemci->{$metot}())
    );
}

Rebuilding the input made one line of the 2020 output make sense. The ounce row read: empty description, sell price 0, empty update time. On first reading you'd think the service quoted a zero sell price for gold. It didn't. If the Satis element is missing from the XML, SimpleXML returns an empty string, and floatval('') is 0. Built with only Kod and Alis, the ounce row reproduced the 2020 output exactly. The library can't tell "no price" from "price is zero". That went into the scaffolding as a bug test too.

Bugs get scaffolded too#

In #2 we said we'd pin those responses down. Confession: on 29 October I looked at the two JSON endpoints in a browser but never saved the responses. This morning, 12 November 2026 (the 10:33 and 10:34 updates), I pulled them again and saved them byte for byte. The difference showed up immediately: the gold endpoint returned 53 items on 29 October, today it returns 54. The service itself made the case for keeping records.

These files aren't input to today's library; the library still speaks SOAP. But the most likely accident in the retrofit is this: the transport changes and the parsing stays as it is. These tests pin exactly that. They pour today's price strings into the old XML shape, run them through the old parser, and record the wrong value that comes out as the expected value:

/** @return array<string, array{string, string, string, float}> */
public static function bugunkuFiyatlar(): array
{
    return [
        'dolar: virgülden sonrası düşer' => ['currency-20261112.json', 'USD', '50,181', 50.0],
        'yen: sıfır olur'                => ['currency-20261112.json', 'JPY', '0,3096', 0.0],
        'has altın: bin kat ucuzlar'     => ['gold-20261112.json', 'HH_T', '6.382', 6.382],
        '500 gr: ikinci noktada kesilir' => ['gold-20261112.json', 'GA500', '3.174.866', 3.174],
        'gümüş: hem binlik hem ondalık'  => ['gold-20261112.json', 'AG250', '19.517,61', 19.517],
    ];
}

In order: the dollar loses everything after the decimal comma, the yen becomes zero, fine gold gets a thousand times cheaper, the 500 g bar is cut off at the second dot, and silver, which uses both separators, ends up as 19.517 instead of 19,517.61.

I did follow Feathers' recipe here for real. For the fine gold test I first wrote the value it should be, 6382. PHPUnit's reply:

1) Altinkaynak\Tests\BeklentiTest::testHasAltin
Failed asserting that 6.382 is identical to 6382.0.

At this stage a red test doesn't say the code is wrong; it says I was wrong about what the code does. The expected value became 6.382 and the test went green. We didn't fix the bug, we named it: tests like this start with testBilinenHata ("known bug"). The day the retrofit switches to the proper price parser from #2, these tests will turn red on purpose, and that day they get updated with a reason. Every BilinenHata test that goes from green to red is a receipt for a fixed bug.

Two more tests belong to the same family:

• Bug: new Altinkaynak('kullanici', '0') never sets the password. The constructor checks with empty(), and in PHP empty('0') is true. An account whose password is zero silently connects with the default credentials.
• Bug: when the service fails, SoapFault escapes as is. The repository has an AltinkaynakException class, but nothing ever throws it. The dead-code finding from #1 is now on record as a test.

Loading the scaffolding#

Green tests don't prove the scaffolding bears load. A test that measures nothing is green too. The way to find out is to break the code on purpose and see whether the tests notice. That's mutation testing. I kept it deliberately small: seven hand-made mutations, the suite runs after each one, then the file is restored.

#!/usr/bin/env bash
# iskele-yuk-testi.sh: src/ altına bilinçli bozulmalar uygula, her birinde
# testlerin kırmızıya döndüğünü doğrula. Yeşil kalan bozulma, iskelenin
# o noktada taşımadığını gösterir. Çıkış 0: hepsi yakalandı, 1: kaçan var.
set -u
cd "$(dirname "$0")"
​
git diff --quiet HEAD -- src/ || { echo "src/ temiz değil; önce değişiklikleri kaydet." >&2; exit 2; }
phpunit >/dev/null 2>&1 || { echo "Testler zaten kırmızı; yük testi anlamsız." >&2; exit 2; }
​
DOSYA=src/Altinkaynak.php
declare -a BOZULMALAR=(
  "floatval yerine intval|s/floatval(\$Kur->Alis/intval(\$Kur->Alis/"
  "alış ile satış yer değiştirir|0,/\$Kur->Alis->__toString/s//\$Kur->Satis->__toString/"
  "GetGold, GetCurrency'yi çağırır|s/ServiceCall('GetGold')/ServiceCall('GetCurrency')/"
  "boş şifre kontrolü kalkar|s/if(!empty(\$password))/if(\$password !== null)/"
  "setter önbelleği sıfırlamaz|0,/\$this->Service = null;/s///"
  "açıklama okunmaz|s/\$Currency->desc\t\t\t= \$Kur->Aciklama->__toString();//"
  "kalemler ters sırada eklenir|s/array_push(\$GetCurrencyResponse->Currencies, \$Currency)/array_unshift(\$GetCurrencyResponse->Currencies, \$Currency)/"
)
​
kacan=0
for satir in "${BOZULMALAR[@]}"; do
  ad="${satir%%|*}"; ifade="${satir#*|}"
  sed -i "$ifade" "$DOSYA"
  if git diff --quiet HEAD -- "$DOSYA"; then
    echo "UYGULANAMADI  $ad"; kacan=$((kacan + 1)); continue
  fi
  if phpunit >/dev/null 2>&1; then
    echo "KAÇTI         $ad"; kacan=$((kacan + 1))
  else
    echo "yakalandı     $ad"
  fi
  git checkout -q -- "$DOSYA"
done
​
echo "---"; echo "kaçan: $kacan / ${#BOZULMALAR[@]}"
[ "$kacan" -eq 0 ]

The seven mutations, in order: intval instead of floatval, buy and sell swapped, GetGold calling the wrong method, the empty-password check removed, a setter that no longer resets the cache, the description no longer read, and items added in reverse order. In the first scaffold the approved-copy test only covered GetMain() and the setter test only covered setServiceURL(). The result (yakalandı = caught, KAÇTI = escaped):

yakalandı     floatval yerine intval
KAÇTI         alış ile satış yer değiştirir
yakalandı     GetGold, GetCurrency'yi çağırır
yakalandı     boş şifre kontrolü kalkar
KAÇTI         setter önbelleği sıfırlamaz
yakalandı     açıklama okunmaz
yakalandı     kalemler ters sırada eklenir
---
kaçan: 2 / 7

Both escaped through the same gap. sed changes the first match; the first parsing loop in the file belongs to GetCurrency(), and the first $Service = null line belongs to setUsername(). My tests were looking at other copies. In #1 we noted that the same parsing code was copied across three methods. That day it was one line in a report; today it turned into two holes. Every copy of copied code needs its own scaffolding. Once the approved-copy test ran over all three methods and the setter test over all three setters through a data provider, the result was kaçan: 0 / 7 (escaped: 0 of 7), exit code 0. In total 18 tests, 26 assertions, and zero lines changed under src/.

Environment: PHP 8.3.6, PHPUnit 9.6.17 (the Ubuntu package). The repository has no composer.json, so PHPUnit was installed system-wide; declaring dependencies is a job for the series finale on inspection and CI. One note: per the PHPUnit 12 announcement, doc-comment annotations like @dataProvider are gone and you use the #[DataProvider] attribute instead. These tests need that update when they move to PHPUnit 12.

Getting an agent to put up the scaffolding#

Having an agent write these tests makes sense; it's dull, repetitive, pattern-shaped work. But without a brief the agent has two easy paths, and both are wrong. The first is to write the value it should be: the test goes red. The second is to see that red and "fix" the code: the test goes green, and the building has changed before the scaffolding was up. That is exactly the next chapter's subject, the floor added without asking.

The brief has to close both paths:

Task: Write characterization tests for this repository. The goal is to record
what the code does today, not to decide whether it is correct.
​
Rules:
- Do not change any file under src/. Only tests/ and phpunit.xml.
- No network. Intercept external service calls with a fake object at a point
  where you can step in without changing the code; name that point and say why.
- Do not guess expected values. Run the test, make the real output the expected value.
- Do not fix behaviour that looks like a bug. Write the test, start its name with
  testKnownBug..., and say in a comment why you think it's a bug.
- Write a separate test for every copied code path.
​
When done: the list of tests you added and the behaviour you couldn't test.

Whichever agent runs it, there are two gates at the end, and both look at exit codes rather than at the agent's word: git diff --quiet HEAD -- src/ returns non-zero if the source changed, and iskele-yuk-testi.sh returns 1 wherever the tests miss a mutation. The brief is written to work unchanged in any agent that can write files, such as Claude Code, Codex or OpenCode; we noted in Five Agents, One Exam that codex exec needs --sandbox workspace-write to write files. I did not run this brief in any agent for this post. I wrote the tests above and I ran them. How well agents follow "don't touch that" rules is what the exam in Illegal Floor will show.

One note on the exam: the starting tree for the five-agent exam in #4 stays exactly as announced. The scaffolding doesn't go into the main branch; it lives on a separate branch.

What scaffolding doesn't do#

It doesn't prove correctness. 8 of the 18 tests protect a known bug. A characterization test doesn't say "this code is right"; it says "this code was like this yesterday too".

It's only as real as its input. The 2020 test runs on a reconstruction. The decimal and date formats match the 2020 output, but the real SOAP response may have had a field, a stray space or an encoding detail I don't know about. The JSON recordings, on the other hand, are real, dated and stored with their SHA-256 hashes.

Its seam is fragile. Writing to a private property through reflection is the price of not touching the source. In the first step of the retrofit the seam should become a client passed in through the constructor; that day this piece of scaffolding gets replaced too.

It's no use for behaviour that changes all the time. Freezing output that changes every day means updating the test every day. What's worth freezing is what must not change: the contract, the format, the order.

So when does it pay off? In any code with no tests, whose behaviour nobody wrote down, that you still have to change. Which is to say, almost every old building.

Scaffolding up, no new floors#

We didn't drive a nail into the building this chapter either; the number of changed lines under src/ is zero. But there are now 18 tests that speak up when something moves, two recorded JSON responses, and a load test showing the scaffolding holds. In the next chapter, Urban Renewal #4: Illegal Floor (28 November), the contractors move in: five agents, the same task, the same repository, and the answer key in a sealed envelope. All chapters live on the Urban Renewal series page.

Scaffolding doesn't make the building any prettier. But when one of the crew drops a brick, it lands on the scaffolding, not on the person walking underneath. Stay with the technology.