How to Validate AI-Generated Code Before Merging

"All tests pass" in an AI agent's summary is a claim, not a result. Agents sometimes run the wrong command, run only the tests near their change, misread the output, or — under pressure to finish — change the test instead of the code. The fix is not to trust the agent less in general, but to make validation something that happens to the agent's work, outside the agent's control, with a hard stop when it fails. Here is a setup that does that.

1. Define "passing" with your own commands

Validation should use the same commands your team and your CI already trust, not a generic check. Most repositories have five kinds:

• install — dependencies from the lockfile (npm ci, composer install, pip install -r requirements.txt)
• lint — style and static rules (eslint, ruff, pint --test)
• typecheck — tsc --noEmit, mypy, phpstan
• test — the full test suite, not a filtered subset
• build — whatever produces the deployable artifact

Find them in your package scripts, Makefile, and CI workflow, write them down per repository, and have a person confirm the list. Run them in that order: there is no point type-checking code whose dependencies failed to install.

2. Make the result a gate, not a report

A red check on a pull request that can still be merged is a suggestion. On GitHub, add your CI jobs as required status checks in the branch protection rules or ruleset for the default branch, so a pull request cannot merge until they pass. Better still, do not open the pull request at all while validation fails — a failing AI pull request in the review queue wastes the reviewer's time and invites "just merge it and fix later".

3. Run the agent's code in isolation

Validation executes code the agent just wrote, so treat it as untrusted. Run it in a throwaway container rather than on a developer machine or a CI runner that holds deployment credentials. With Docker, the important controls look like this:

docker run --rm \
  --network none \
  --read-only --tmpfs /tmp:size=256m \
  --cap-drop ALL --security-opt no-new-privileges \
  --cpus 2 --memory 2g --pids-limit 512 \
  -v "$PWD":/work -w /work \
  node:22 npm test

• No network after dependencies are installed. Tests that need the internet are flaky anyway, and a compromised dependency cannot phone home.
• Read-only everywhere except the working copy and a small temporary directory.
• No capabilities, no privilege escalation, and limits on CPU, memory, processes, and time, so a runaway loop fails instead of hanging.
• No real secrets. Pass only test values the suite needs. Never production credentials, and never your platform's own tokens.

If your validation runs in GitHub Actions, the same principle applies: set minimal permissions for the workflow's token, keep deployment secrets in protected environments, and be careful with pull_request_target, which runs with access to secrets.

4. Protect the tests from the agent

The easiest way for an agent to make a failing test pass is to change the test. Watch for it:

• Flag any change under your test directories in review, and require code-owner review for them.
• Look for deleted tests, newly skipped tests, loosened assertions, and snapshot files regenerated wholesale.
• If you track coverage, fail the check when it drops.
• Tell the agent explicitly, in its instructions, that changing existing tests requires a reason stated in the plan.

5. Close the loop — with a limit

When a check fails, feed the failing command's output back to the agent and let it try again. Agents fix a lot of lint and type errors this way. But cap the attempts: two retries is a sensible default. After that, stop and hand the work to a person with the diff and the failing output, rather than letting the agent keep rewriting until something passes.

6. Be honest about repositories without tests

Some repositories have no test suite. Validation there can still run install, lint, typecheck, and build, but it is weaker, and reviewers should know that. Mark those changes clearly — for example with a "reduced validation" note in every pull request description — instead of letting a green check imply more than it proves.

A checklist

• Validation commands come from the repository and were confirmed by a person.
• The full suite runs, not just the tests near the change.
• Failing validation blocks the pull request or the merge — in code, not by convention.
• The agent's code runs in an isolated container with no network after install and no real secrets.
• Changes to tests are flagged and need an owner's review.
• Fix attempts are capped, then a person decides.
• The pull request says which commands ran.

How Dualzo handles this

Dualzo proposes validation commands from your scripts, Makefile and CI, has an admin confirm them, and runs them after every task in an ephemeral, locked-down container. Failures go back to the model for up to two fix attempts, and a pull request cannot be opened while validation is failing. Read more about sandboxed validation in Dualzo.