The useful question is not only whether an agent performs well on a public benchmark. It is whether a specific model, prompt, tool set, and budget reliably resolves failures that actually resemble your engineering work.

RepoTrials turns fixes already present in Git history into private, repeatable coding-agent evaluations.

A concrete proof-of-value loop

A repository already records behavioral transitions:

1. a base revision passed its original tests;
2. a new test exposed a bug;
3. a human patch fixed that behavior; and
4. protected tests continued to pass.

RepoTrials reconstructs the historical state, separates verifier material from the agent workspace, and runs every agent against the same sealed task.

For a base revision B, hidden test patch T, and historical solution S, validation requires:

BASE   B       + original tests   → pass
RED    B       + T                → relevant failure
GOLD   B + S   + T                → pass
NOOP   B       + T                → not resolved

Agent grading is behavioral. A candidate does not need to reproduce the human diff, but it must make the fail-to-pass tests green without breaking protected regression tests.

The smallest useful experiment

The v0.1.0 demo builds a real two-commit repository and compares two command-based agents:

• a no-op agent that leaves the failure unresolved;
• a fixing agent that edits the historical workspace correctly.

git clone --depth 1 --branch v0.1.0 https://github.com/PozziTiv4ik/Repo-Trials.git
cd Repo-Trials
python -m pip install -e ".[dev]"
python scripts/demo.py

It produces an auditable comparison:

noop-agent   0/1 resolved
fix-agent    1/1 resolved
delta       +100 percentage points

No model API key is required for this demo. The same pipeline mines tasks, validates BASE/RED/GOLD transitions, runs command-based agents, stores machine-readable results, renders HTML reports, and exports tasks for Harbor.

Why private benchmarks matter

Shared benchmarks are valuable for ecosystem-wide comparisons. Repository-specific tasks answer a different set of operational questions:

• Did the new model improve on our own failure modes?
• Is a cheaper configuration good enough for routine fixes?
• Does a prompt change help across repeated trials?
• Can another engineer reproduce the result later?

RepoTrials is local-first. Source snapshots, hidden tests, reference patches, and run artifacts stay under the operator-controlled .repotrials/ directory by default.

Current boundaries

v0.1.0 is a first public release, not a security or procurement certification. Human task review remains necessary. The strongest current path is a Python repository with reproducible historical tests. Repository tests and agent commands execute arbitrary code, so untrusted workloads require isolation.

The evaluation contract, methodology, threat model, and schemas are public precisely so teams can inspect the evidence instead of trusting a single score.

Try the v0.1.0 release, read the full launch article on DEV, and tell us which repository setup fails first.

Disclosure: this article was drafted with AI assistance and checked against the v0.1.0 project documentation.