Guides

Evals & quality

A code reviewer's value is judgment quality against the tenant repo's own standards, and its main failure mode is social: if quality slips, humans mute the agent. Nothing else in the system measures review quality — only mechanics — so the eval harness is a first-class deliverable, not test scaffolding.

The golden corpus

Each tenant maintains a corpus of curated "golden PRs" with known expected findings, under packages/evals/corpus/<tenant>/<case>/:

• Defect cases in the tenant's own failure categories — for Helix: cross-variant misses, RBAC gaps, hardcoded secrets, Docker-test violations, design-system violations, correctness bugs.
• Clean-PR controls whose expected finding count is zero — the false-positive control. A reviewer that flags noise on clean PRs gets muted just as fast as one that misses bugs.

Each case declares its scope key, changed paths, review tier, and expected findings (category, file, a lineWindow the finding must land in, severity). Scoring is deterministic-first: findings are closed-enum structured output, so precision/recall per category is computed exactly — right category, right file, line inside the window.

Baselines and the regression gate

npm run evals                        # offline: MockRunner + MockJudge
npm run evals -- --tenant helix      # one tenant
npm run evals -- --update-baseline   # rewrite baselines/<tenant>.json

Every run is scored against a recorded baseline per tenant. The gate fails (exit 1) only on regression against that baseline — never on absolute thresholds. This matters: quality bars are earned by measurement, not declared; and a deliberately imperfect baseline is allowed to stay imperfect until someone improves it knowingly (--update-baseline is a reviewed, committed change).

No model, prompt, or ruleset change ships without a recorded eval run. This is also the swap gate for provider portability: "try a cheaper model" is a benchmark run plus a config change, never a leap of faith.

Scorer integrity

The corpus intentionally contains cases whose baseline is less than perfect — not accidents, but tests of the scorer itself:

Live vs mock runs

npm run evals                # MockRunner: deterministic, offline, CI-safe
npm run evals -- --live      # AgentSdkRunner: real model; ADO stays faked

Mock runs exercise the full pipeline — rules routing, prompt assembly (prompt_sha recorded in the report), tier selection, scoring — with deterministic outputs, so they run in CI with no keys. Live runs swap in the real Claude Agent SDK runner (an inference endpoint is required) while ADO remains the in-memory fake: a live eval can never post to a real PR. Reports land in packages/evals/reports/.

Opik integration

npm run evals -- --opik      # or ARGUS_OPIK=1

Each run can additionally be exported to Opik, Comet's open-source LLM-eval platform — one dataset per tenant (argus-golden-<tenant>, items keyed stably by tenant/case and synced idempotently) and one experiment per tenant per run, with a trace per case and precomputed feedback scores (precision, recall, F1, detail score, clean-control violations, injection-flag correctness) plus prompt_sha and model id in the experiment config.

Three properties keep it safely optional:

• The gate stays local. Opik is observability; the regression gate never depends on it.
• Failures warn and continue. An unreachable server or bad key never fails an eval run.
• Off means off. Without the flag the Opik SDK is never even imported — offline CI touches no network.

For data residency, self-host Opik (docker compose from the Opik repo) and point OPIK_URL_OVERRIDE at it; see the Opik variables in the configuration reference.