Gate AI agent regressions on every pull request.
Drop agentdiff-check into your CI pipeline. Diff candidate execution graphs against golden baselines, post rich diagnostic blame trees to PR comments, and lock the merge button if thresholds fail.
Drop into any repo in 30 seconds.
Execute your agent test script in CI, record the candidate trace, and invoke agentdiff-check. If divergence, cost, or retry loop thresholds breach tolerance, the action fails with exit code 1.
name: AgentDiff Gate
on: [pull_request]
jobs:
gate:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Record Candidate Trace
run: |
pip install agent-trajectory-diff
agentdiff record my_agent:run \
--input '{"scenario": "customer_support"}' \
--output traces/pr_candidate.json
- name: Gate Against Golden Baseline
uses: lostmartian/agentdiff/.github/actions/agentdiff-check@main
with:
baseline: baselines/golden_run.json
candidate: traces/pr_candidate.json
pr: ${{ github.event.number }}
github-token: ${{ secrets.GITHUB_TOKEN }}What happens on every pull request.
Three deterministic safeguards run on every push, giving your team actionable feedback before bad code hits main.
Rich PR Diagnostic Commentary
Posts an interactive markdown summary directly to the PR with exact divergence ratios, step-by-step diff trees, token spikes, and isolated culprit root-cause lines.
Exit Code 1 Merge Blocker
Emits strict non-zero exit codes upon regression, automatically failing the GitHub check suite and preventing engineers from merging broken agent behaviors.
Automated Baseline Rotation
Set update-baseline = true to safely advance your golden baseline trace only when the candidate execution passes all assertion criteria.
Did the PR pass — or did the gate move?
When developers weaken test thresholds (e.g. raising max_divergence from 0.25 to 0.75 to force a broken agent to pass), AgentDiff compares the PR configuration against the main branch baseline config.
Threshold alterations are flagged as high-priority security warnings in the PR review timeline, preventing silent regression masking.
- name: Fetch Baseline Policy from Main
run: git show origin/main:agentdiff.toml > /tmp/baseline-agentdiff.toml
- name: Gate Against Baseline with Governance Guard
uses: lostmartian/agentdiff/.github/actions/agentdiff-check@main
with:
baseline: baselines/golden_run.json
candidate: traces/pr_candidate.json
baseline-config: /tmp/baseline-agentdiff.toml # Flags loosened thresholds
pr: ${{ github.event.number }}
github-token: ${{ secrets.GITHUB_TOKEN }}Complete Action inputs specification.
Everything configurable in agentdiff-check:
| Parameter | Requirement | Description |
|---|---|---|
| baseline | Required | Path to the committed golden baseline trace JSON in your repository. |
| candidate | Required | Path to the candidate trace JSON generated by the agent during the current PR workflow run. |
| pr | Optional | Pull request number (e.g. ${{ github.event.number }}) to post the formatted divergence report and culprit blame tree to. |
| baseline-config | Optional | Path to the main branch agentdiff.toml. Enables the Goodhart guard to detect if PR authors silently weakened tolerance thresholds. |
| adapter | Optional | Telemetry parser: auto (default), langgraph, crewai, openai_agents, openinference, langfuse, langsmith, generic. |
| update-baseline | Optional | When true, automatically advances the stored golden baseline trace if the PR trajectory verdict passes all quality gates. |
| github-token | Optional | GitHub secret token (e.g. ${{ secrets.GITHUB_TOKEN }}) with pull-requests: write permission for automated commentary. |
Record. Diff. Gate. Ship with confidence.
Add one GitHub Action step to automate trajectory regression testing across your engineering team in under 5 minutes.