Same Exam for Five Agents: The Answer Key Stays Sealed

This week a LinkedIn message landed in my inbox. It came from Rüzgar İmren, co-founder and CTO of Atlantic. He described their product, Factory, in a few lines: you describe the work, it writes and runs the code on its own isolated machine, tests it and opens a PR, and the merge decision stays with your team. He asked me to try it with my team and tell him what I think.

Trying one product and saying "liked it" or "didn't" is easy. More useful is giving the same job to five agents: Claude Code, Codex, Antigravity, Qwen Code and Atlantic Factory. But an AI coding agent comparison starts with rules, not with a results table. There are no results in this post. I'm writing down the exam rules, the scoring and the ways to cheat before I see a single outcome, so that nobody, me included, can bend the rules toward the winner later.

Why rules first?#

Whoever writes an exam puts the answer key in a sealed envelope, and the envelope is opened after the exam. Not because anyone doubts the teacher. Once the papers are in, it becomes very easy to argue which question was "actually easy" and which answer "deserves half credit".

That is the problem I see most in benchmark posts. The tester sees the results, then builds the criteria. Whatever the winner is good at gets more weight; whatever the loser is good at becomes "out of scope". No bad intent required, the brain does it on its own. The fix is old: publish the scoring in advance.

The candidates#

The five are not the same kind of tool, and that is the first problem. Four are terminal agents you drive. One is a pipeline that walks from an issue to a PR by itself.

A few details from their own docs (October 15, 2026), because each one becomes a line in the protocol:

• Codex: run without flags, codex exec starts in a read-only sandbox. It needs --sandbox workspace-write to edit files. The --full-auto flag from older scripts is no longer supported.
• Claude Code: the --output-format json output includes total_cost_usd, and the docs themselves say it is a client-side estimate that can differ from your bill.
• Antigravity: according to a June 10, 2026 post on the Google Cloud blog, the Go-based CLI is the surface to pick for headless work such as SSH sessions or remote containers. The model in use gets recorded on test day.
• Qwen Code: adapted from Google's Gemini CLI v0.8.2; it works with OpenAI, Anthropic, Gemini and Qwen APIs, and with local models. The catch: point Qwen Code at another vendor's model and you are testing that model in Qwen's shell, not Qwen. The protocol pins a Qwen model.
• Atlantic Factory: per the product page, you install a GitHub App, work starts from a ticket, the branch boots on a separate machine from a factory.deploy.yml, a browser agent tries the app (the page's example reads "23 of 40 steps"), Factory reviews its own diff for up to two rounds and opens a PR. It never pushes to the default branch. No model names are given, only tiers from Low to Max; the security page says calls go through Amazon Bedrock to Anthropic Claude and OpenAI GPT models.

So weighing Atlantic on the same scale as the terminal agents would be comparing apples and oranges. I keep two separate results. The first is the code itself, scored the same way for all five. The second is the process: how many times a human had to touch the work between the task and a review-ready PR. That one is reported, not scored.

The question paper: which repo, which task?#

While I was talking the idea through, ChatGPT suggested building a small e-commerce app from scratch and having the agents add stock reservation. Reasonable, but I won't. A repo I write for the exam gets shaped by my expectations; its bugs would be planted, not grown.

Instead I'm using a real old building: the Altınkaynak exchange rate library I wrote in 2020 (in Turkish; code at github.com/HalitYesil/altinkaynak). Ten PHP files, last commit September 6, 2020, wired to a SOAP service. Altınkaynak's web services page now lists only two addresses, both returning JSON over HTTPS. The task comes straight from that gap. The agents receive it in Turkish; this is the translation:

Task: Move the library to Altınkaynak's current JSON endpoints.
Source: https://altinkaynak.com/Araclar/Servisler
- Keep the signatures of the existing public methods that return currency and gold lists.
- Remove the SOAP dependency and the credentials in the source code.
- Add tests that verify the change.
- Do not touch files unrelated to the task.
When done, list the files you changed and why in the PR description.

The four terminal agents get the text on the command line, Atlantic gets it verbatim as a GitHub Issue. The repo has no AGENTS.md, CLAUDE.md or any other rules file. In Vibe Coder 4.2 we saw that tools read the same file differently; this exam doesn't measure that, everyone sees only the task text.

The rules#

1. Same start. Five separate repos are generated from a template; GitHub's template repository feature gives each one an unrelated history, so PRs don't get mixed up. Commit IDs will differ, but the hash of the file tree must match. The script below checks that before the exam starts.
2. Same task text, the block above. Not a word changes.
3. No intervention. No human touches the first round. If an agent asks a question, there is one canned answer: "Continue with the information in the task text." Every question is counted.
4. 60-minute ceiling. When time runs out, whatever exists is scored.
5. Web search off. Some of these agents can search the web, and this post is online the moment it goes live. Where a tool can't turn it off, that goes on record.
6. Model and version on record. Tool version and model for every run; the tier for Atlantic.
7. Hidden test suite. Tests the agents never see run on a clean checkout before anything is merged. They get published with the results, not today. The answer key stays in the envelope.
8. At least two runs. If a tool gives different results across runs, both get reported. No picking the better one.
9. Cost on its own line. Each tool's self-reported amount, with its source. A subscription and an API bill are not the same thing and won't be mixed.

Here is the script for rule one. It compares the tree of each repo's root commit with the template's tree:

#!/usr/bin/env bash
# sinav-yoklamasi.sh: does every agent's repo start from the same tree?
set -euo pipefail
​
BEKLENEN=${1:?Kullanım: sinav-yoklamasi.sh <şablon-ağaç-hash> <depo>...}
shift
HATA=0
​
for DEPO in "$@"; do
  KOK=$(git -C "$DEPO" rev-list --max-parents=0 HEAD | tail -n 1)
  AGAC=$(git -C "$DEPO" rev-parse "$KOK^{tree}")
  if [ "$AGAC" = "$BEKLENEN" ]; then
    echo "tamam  $DEPO  ${AGAC:0:12}"
  else
    echo "FARKLI $DEPO  ${AGAC:0:12}" >&2
    HATA=1
  fi
done
​
exit "$HATA"

I ran it against a copy of altinkaynak. The template tree is 7c6df0b4e980. With five repos built as if generated from the template, each with its own history, all five printed tamam ("ok") and the exit code was 0. After adding a single line to src/Altinkaynak.php in one of them and rebuilding it, that repo printed FARKLI ("different") with e16266be68cd and the exit code was 1. One line of difference and the exam doesn't start.

Scoring, before the results#

Not scored but reported: total time, cost, the number of questions the agent asked, and the process result for Atlantic.

Why 15 for scope: I think the most expensive mistake a coding agent makes is not wrong code but a file it touched without asking. Reviewing a correct PR that touches thirty files can take longer than fixing a wrong PR that touches three.

The fine print on the invitation#

Two disclosures. First: I'm not taking any payment or perks from Atlantic, and I'm not using the referral link from the message. Here is the product address directly: factory.getatlantic.ai. Rüzgar İmren and Ege Mustafa Çelik founded the company in March 2026; in June, Webrazzi reported (in Turkish) that they raised from Galata Business Angels at a 5 million dollar valuation.

Second, the card. The message said "free, no card". The pricing page today says "Start with $50 free for 7 days", while the FAQ on the same page says "You save a card during onboarding and pay $0 today." When I first asked ChatGPT to review the message it noted the saved card; in its second answer it corrected itself to a card-free start. Today's page sides with the first answer. So the trial terms in the results post will come from a screenshot taken on test day, not from memory.

On code handling, the security page is direct, and I appreciate that. Workers run in AWS us-east-1, and model calls can be served in another AWS region through Bedrock's cross-region inference. Tests run in containers with networking turned off. QA screenshots and browser recordings sit in S3 behind unlisted links, are deleted after 180 days, and until then "Anyone holding a link can open it." Fine for a public repo. If you plan to try it on client code, read that sentence and decide.

Before the test I'll ask Rüzgar three things: a sample factory.deploy.yml for a PHP project, whether the agent's web access can be turned off, and the name of the model used in a run. The answers go into the results post.

When does the envelope open?#

The results will be published in Kaçak Kat ("The Illegal Floor"), the fourth chapter of the Urban Renewal series that starts this Saturday, on November 28: five PRs, the diffs, the hidden tests and their output, all open for anyone to check.

The envelope is sealed. The answer key picks the winner, not me. Stay with the technology.