agentdiff
/Regression Gates

Regression Gates

AgentDiff lets you define thresholds so a change either passes or blocks in CI. There are two ways to gate: the assert_no_regressions helper in Python, and the CLI's --fail-on-regression flag.

Python: assert_no_regressions

python
from agentdiff.testing import assert_no_regressions assert_no_regressions( report, max_divergence=0.25, # max TDI (default 0.25) max_cost_increase_pct=5.0, # max cost increase %, default 5.0 allow_loops=False, # reject any detected loop max_wasted_effort=0.10, # max WEI, default 0.10 max_recovery_step_ratio=1.5, # opt-in: max RSR (None = disabled) )

Raises an AssertionError naming each violated threshold if any metric exceeds its boundary:

text
AssertionError: AgentDiff Regression Verification Failed: - Trajectory Divergence Index (TDI) of 0.3333 exceeded threshold of 0.2500. - Candidate Wasted Effort Index (WEI) of 0.2500 exceeded threshold of 0.1000.

CLI: --fail-on-regression

bash
agentdiff baseline.json candidate.json --fail-on-regression

Exits with code 1 when a regression is detected. The default thresholds are:

FlagDefaultMeaning
--max-divergence0.3Max TDI before regression.
--max-loops0Max loop count before regression.
--max-cost-delta10.0Max cost increase % before regression.
--max-recovery-ratio(off)Max Recovery Step Ratio before regression (opt-in — the gate is disabled unless set).
bash
agentdiff baseline.json candidate.json \ --fail-on-regression \ --max-divergence 0.2 \ --max-loops 1 \ --max-cost-delta 5.0 \ --max-recovery-ratio 1.5

These defaults can also be committed in an agentdiff.toml (see Configuration) - explicit flags still win.

pytest plugin

The same gating is available as a pytest plugin:

bash
pytest --agentdiff --agentdiff-max-divergence 0.2

See the pytest Plugin guide.

Tuning the thresholds

There's no universal "right" value. Start conservative (low max_divergence, allow_loops=False) and loosen as you learn what changes are intentional. Use --max-drift and baseline rotation to control how often the baseline advances instead of letting gates creep silently - see [Baseline Rotation].