Packages

@argus/evals

The golden-PR eval harness (PRD §6.5). Nothing else in the system measures review quality — only mechanics — and a reviewer whose quality slips gets muted by humans. This package makes quality measurable per tenant and gates every prompt/model/rules change on regression against a recorded baseline.

Place in the system

Depends on @argus/worker (it runs the same pipeline pieces as production — prompt assembly, tier selection, runners) and @argus/rules. bin/argus-evals.ts → main() is the CLI; reports land in packages/evals/reports/.

The run, end to end

flowchart TD
  corpus["corpus: packages/evals/corpus/tenant/case<br/>fixture repo + diff + expected findings<br/>(incl. clean-PR false-positive controls)"] --> fx["FixtureCheckout + FakeAdoClient<br/>(evals never touch real ADO)"]
  fx --> rules["routed rules + assembled prompt<br/>(prompt_sha recorded)"]
  rules --> runner{"runner"}
  runner -- default --> mock["MockRunner<br/>deterministic, CI-safe"]
  runner -- "--live" --> live["AgentSdkRunner<br/>real model, fake ADO"]
  mock --> validate["validated FindingsOutput"]
  live --> validate
  validate --> score["deterministic scoring:<br/>category + file + lineWindow<br/>precision / recall per category<br/>+ judge over detail prose"]
  score --> gate{"regression vs<br/>baselines/tenant.json?"}
  gate -- yes --> fail["exit 1"]
  gate -- no --> ok["reports (JSON + markdown)<br/>+ optional Opik export"]

Figure 1 · An eval run end to end — fake ADO throughout, deterministic scoring, and a gate that fails only on regression against the baseline.

Flags (all real — see bin/argus-evals.ts): --tenant, --corpus, --baselines, --out, --live, --update-baseline, --opik.

What keeps the scorer honest

The gate fails only on regression against the recorded baseline, never on absolute thresholds — bars are earned by measurement.
The corpus contains deliberate sub-100% cases (missed-finding, near-miss-line, false-positive-noise) that test the scorer itself; a perfect score across the board is a scorer bug.
Clean-PR controls (expected findings: zero) count noise against precision.

Key exports

main() / parseArgs() (cli.ts), runCase/scoreCase/aggregateTenant (pipeline.ts, score.ts), MockRunner (fixture-driven), MockJudge + the DetailJudge interface, baseline load/compare/update (baseline.ts), report writers (report.ts), and the opt-in opikSink.ts (lazy import — with the flag off, the Opik SDK is never loaded).

How it's tested

The harness is itself heavily tested: e2e.test.ts (full runs, gate behaviour), productionParity.test.ts (evals assemble the same prompt the worker does — the drift that would silently invalidate every score), score.test.ts, baseline.test.ts, corpus.test.ts, diff.test.ts, opikSink.test.ts — offline throughout.

← db docs-server →