Cloudflare's Security Audit Skill: The Hunter Finds, the Devil's Advocate Refutes

In Vibe Coder #5.3, published on Tuesday, I lined up the model-based security reviewers from four ecosystems and gave them all the same label: an eye. An eye, not a lock. Which leaves an awkward question. Who checks what the eye saw? When a model says it found a vulnerability, is the same model also deciding that the vulnerability is real?

This week's radar entry is one company's answer, pulled out of its own production work: Cloudflare's security-audit skill. It is an instruction package you install into your coding agent, and it turns that agent into a six-phase security auditor. The most valuable line in it is not an attack list. It is a rule of procedure: every finding gets a lawyer first, and that lawyer's job is to tear it apart.

So this week's metaphor is the devil's advocate. The hunter finds, the advocate tries to refute, and only the case that survives goes to a verdict.

What is the security-audit skill?#

In one sentence: an MIT-licensed Agent Skill that walks a coding agent, through isolated sub-agents, from reconnaissance to hunting, validation, structured output, independent record verification and reporting.

The repository is cloudflare/security-audit-skill. The main author in the commit history is Dan Jones (literally-dan), listed on his Cloudflare blog author page as Principal Red Team Engineer. The skill is the starting point described in Build your own vulnerability harness, published on 18 June 2026 by Dan Jones, Alexandra Godoi and Grant Bourzikas. In the README's words, Cloudflare's fleet-wide vulnerability discovery harness grew out of this single-repo skill. The harness itself is not public; the post says "hopefully the harness itself will follow shortly."

Installing it#

Installation goes through Vercel's Skills CLI (MIT), and the same command targets different agents with -a:

# Claude Code (.claude/skills/)
npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit -a claude-code
​
# Codex, Cursor, OpenCode, Gemini CLI and Antigravity use .agents/skills/ inside the project
npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit -a codex
​
# user-level install
npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit --global

Then you tell the agent "security audit this codebase". The skill has two modes. Security questions and focused reviews stay in guidance mode and write no files. Ask explicitly for a full audit and all six phases run, with output going outside the repository by default, to ~/security-audit-skill/<repo-name>/run-<N>.

Let me be upfront: for this piece I did not install the skill into an agent and run an audit, and I did not run its Node scripts. I read the source at commit c1c8a8c (14 September 2026) line by line. The PHP gate further down I wrote and ran myself.

The hunter finds, the advocate refutes#

The six phases hang off one rule: the agent that finds a candidate never validates it. In phase three every candidate goes to a fresh agent that has never seen it, and that agent's prompt opens with:

You did not write this candidate. Try to refute it from repository source and bounded local evidence.

The verifier is not shown any other verifier's conclusion either. In phase five the final records are checked against source once more, again by fresh agents.

The harness post explains why the rule exists in a single line: "If a Hunter is allowed to grade its own homework, it will confidently validate everything it outputs." The advocate's work is measured, too. After better context injection from the reconnaissance phase, the initial validation rejection rate dropped from 40% to 11%. That improvement is only visible because the advocate sits apart; a hunter grading itself would never produce a rejection rate at all.

The same post has a line that lands right on top of the taint tracker from 5.3: "We plumbed Semgrep all the way through, and the Hunters invoked it zero times in a month of runs." The most-used tool turned out to be the wishlist, the queue where agents write down what they would need to confirm something. It has been written to 25,472 times across 128 repos.

Hidden gem: every output record carries one of three verdicts. confirmed needs a complete source trace and a bounded local observation, and it is the only verdict that carries a severity (overall_severity). needs_validation is not a "low-confidence finding": it is a source-grounded candidate whose decisive fact cannot be seen in the repository or the sandbox. It has no severity and it must list its blockers. rejected is a refuted candidate with its reason. The schema enforces this; you cannot put a severity on a needs_validation record.

Wiring findings.json to a gate#

Remember the lesson from 5.3: a model reviewer is an eye, the lock is a command that stops on an exit code. Because this skill's output is machine-readable JSON, building that lock is easy. The script below reads findings.json, exits with 2 if there is a confirmed finding at or above the threshold, and lists the pending validations as work for someone.

The second check matters more. An empty findings.json does not mean clean code; the run may have stopped halfway when the budget ran out. The skill records that in run_status inside run-metadata.json, and it only writes complete once every candidate has an independent final disposition. The gate does not open without complete.

<?php
​
declare(strict_types=1);
​
/**
 * Audit gate: reads the findings.json written by the security-audit skill.
 *   php audit-gate.php <findings.json> [threshold]   threshold: low|medium|high|critical (default high)
 * Exit: 0 gate open, 2 confirmed finding at/above threshold or run not complete, 1 file/format error.
 * needs_validation records do not close the gate, but are listed as work for their owner.
 */
​
const RANK = ['informational' => 0, 'low' => 1, 'medium' => 2, 'high' => 3, 'critical' => 4];
​
$file = $argv[1] ?? '';
$threshold = $argv[2] ?? 'high';
if ($file === '' || !is_file($file) || !isset(RANK[$threshold])) {
    fwrite(STDERR, "Usage: php audit-gate.php <findings.json> [low|medium|high|critical]\n");
    exit(1);
}
​
try {
    $records = json_decode((string) file_get_contents($file), true, 64, JSON_THROW_ON_ERROR);
} catch (JsonException $e) {
    fwrite(STDERR, "Cannot read findings.json: {$e->getMessage()}\n");
    exit(1);
}
if (!is_array($records) || !array_is_list($records)) {
    fwrite(STDERR, "findings.json must be an array.\n");
    exit(1);
}
​
// An empty list is not "clean": if the run stopped halfway, the count means nothing.
$meta = dirname($file) . '/run-metadata.json';
$status = is_file($meta) ? (json_decode((string) file_get_contents($meta), true)['run_status'] ?? '?') : 'missing';
if ($status !== 'complete') {
    fwrite(STDERR, "run-metadata.json run_status: $status. A partial run does not open the gate.\n");
    exit(2);
}
​
$count = ['confirmed' => 0, 'needs_validation' => 0, 'rejected' => 0];
$blocking = [];
$pending = [];
​
foreach ($records as $r) {
    $verdict = $r['verdict'] ?? '';
    if (!isset($count[$verdict])) {
        fwrite(STDERR, "Unknown verdict: " . var_export($verdict, true) . "\n");
        exit(1);
    }
    $count[$verdict]++;
    $trace = is_array($r['trace'] ?? null) ? $r['trace'] : [];
    $last = end($trace);
    $where = is_array($last) ? "{$last['file']}:{$last['line']}" : '?';
​
    if ($verdict === 'confirmed') {
        $severity = $r['severity']['overall_severity'] ?? '';
        if (!isset(RANK[$severity])) {
            fwrite(STDERR, "{$r['fingerprint']}: confirmed record without a severity.\n");
            exit(1);
        }
        if (RANK[$severity] >= RANK[$threshold]) {
            $blocking[] = "[$severity] {$r['title']} ($where)";
        }
    } elseif ($verdict === 'needs_validation') {
        $pending[] = "{$r['title']} ($where) — " . implode('; ', $r['blockers'] ?? []);
    }
}
​
printf("confirmed %d, needs_validation %d, rejected %d\n", $count['confirmed'], $count['needs_validation'], $count['rejected']);
foreach ($pending as $line) {
    echo "Needs validation: $line\n";
}
if ($blocking !== []) {
    foreach ($blocking as $line) {
        fwrite(STDERR, "At/above $threshold: $line\n");
    }
    exit(2);
}
​
echo "No confirmed finding at/above $threshold.\n";
exit(0);

I tried it against a hand-written three-record sample using the schema's field names (one confirmed, one pending, one rejected; it does not carry every field the schema requires) and two fake run folders (PHP 8.3.6 CLI, 10 November 2026):

$ php audit-gate.php run-1/findings.json
confirmed 1, needs_validation 1, rejected 1
Needs validation: PHP may execute in the upload folder (src/Upload.php:27) — whether Apache runs PHP under uploads/ is not in the repository
At/above high: [high] Order detail page shows another customer's order (src/OrderRepository.php:41)
$ echo $?
2
$ php audit-gate.php run-2/findings.json
run-metadata.json run_status: incomplete. A partial run does not open the gate.
$ echo $?
2

The second run's findings.json is an empty array. Without the run_status check the gate would have opened with 0. Raise the threshold to critical and the first run returns 0 as well. This belongs in a nightly or pre-release job rather than on every commit; the pipeline side is the subject of chapter 6.1 in the series.

When it does not help#

• One run is half a job. The README says so itself: in their test runs, a single run found roughly half of the vulnerabilities that repeated runs found in total. The harness post is equally plain: "We don't claim a false-negative rate for our system."
• It is not cheap. A full audit opens with four reconnaissance agents, adds a coverage critic after each hunting wave, and spends one or two verifiers per candidate. It needs an agent that can run parallel sub-agents. You can cap the budget as a number of agent invocations; there is no dollar estimate.
• No sandbox, everything waits. Running target code requires an OS-enforced sandbox with no external network, an empty allowlisted environment, writes only to a scratch directory, and resource limits. Without it the skill does not execute anything and leaves the candidate as needs_validation. On a machine without those controls, every candidate that needs execution to confirm stays in that column.
• Nothing PHP-specific. The attack classes are language-neutral; none of the eleven companion files mentions PHP, Composer or any PHP framework.
• No releases. There is not a single release or tag; npx skills add installs whatever the default branch holds at that moment. Write down which commit you installed.
• It is not just Markdown. The package includes two validator scripts that run on Node. What to do before installing a skill was the subject of the SkillSpector piece, and it applies here too.

Maintenance status (10 November 2026)#

Alternatives#

The other four were read from their official docs on 8 November 2026; links are in 5.3. The split is this: they are a quick eye over a diff or a repository. Cloudflare's is a slow, expensive procedure designed to be run again and again.

Where I would use it#

Not on every commit. The mine gate from 5.3 already sits in the agent's session; this skill belongs before a large change. In a setup like the one from the multi-storefront piece, several storefronts on one database, a scoped run that selects only the data-isolation companion is where it would pay off most. The first rule in that file is almost word for word the thesis of that piece: a tenant or owner field on a record is not isolation; isolation is the query, key or policy that enforces it on every read and write path.

One more caveat: the skill never probes a live environment. If your production Apache config, proxy behaviour or provider settings are not in the repository, the verdict is needs_validation. The tool is honest about what it cannot see. Closing that column is your job.

The hunter's speed is impressive, but the advocate is what makes this package worth installing. Don't let an agent grade its own homework. Stay with technology.