Open source · Early alpha

Stop guessing.

Compare agent trajectories.

Static string assertions cannot evaluate non-deterministic agents. AgentDiff maps and compares execution traces as DAGs in CI/CD — automatically quantifying trajectory drift, redundant tool loops, and resource regressions.

Star on GitHub Read the docs

Works with

Generic JSONDeepEvalOpenInferenceLangfuse

Workspace preview

Interactive trajectory blueprint

Select a diagnostic scenario below to inspect how AgentDiff aligns execution graphs and exposes regressions side-by-side.

Baseline run

Diff

Candidate run

Step 01 · 120 tokens · $0.0002

Agent start

Aligned

Step 01 · 120 tokens · $0.0002

Agent start

Step 02 · 320 tokens · $0.0005

list_tables

Aligned

Step 02 · 320 tokens · $0.0005

list_tables

Step 03 · 480 tokens · $0.0009

read_schema

{ table: 'users' }

Cached

—

Step 04 · 480 tokens · $0.0009

read_schema

{ table: 'orders' }

Cached

—

Step 05 · 600 tokens · $0.0015

execute_sql

SELECT * FROM orders LIMIT 10;

Aligned

Step 05 · 600 tokens · $0.0015

execute_sql

SELECT * FROM orders LIMIT 10;

Step 06 · 400 tokens · $0.0008

Synthesize

Aligned

Step 06 · 400 tokens · $0.0008

Synthesize

Gating status

PassThe candidate run reduces redundant database queries by caching database schema requests, shortening the execution graph.

Evaluation logs

Cost delta-62.5%

Latency delta-1.8s

WEI threshold0.15 Index

NormalizationOpenInference

Capabilities

Engine specifications

01. Divergence index

Trajectory divergence

Quantifies structural execution difference using graph alignment algorithms. Detects when model upgrades lead to fully divergent tool paths.

02. Wasted effort

Wasted effort index

Calculates ratios of redundant or abandoned tool execution steps, flagging prompt modifications that cause unproductive token consumption.

03. Loop buster

Loop buster analysis

Exposes cyclical tool calling patterns where an agent repeatedly queries the same endpoint with identical parameters without state progress.

04. Automation

CI/CD regression block

Standardized JSON export format integrates natively into GitHub Actions and GitLab pipelines. Fail builds when cost drift or loops cross limits.

Developer integration

Zero boilerplate DX

AgentDiff exposes a programmatic Python SDK tailored for testing frameworks like `pytest`, alongside a strict CLI runner for automated pipeline integration.

Supported formats

Generic JSONDeepEvalOpenInferenceLangfuse

import pytest
from agentdiff import load_trace, compare
from agentdiff.testing import assert_no_regressions

def test_agent_refactor_efficiency():
    # load_trace() auto-detects format: generic, deepeval, openinference, langfuse
    baseline  = load_trace("tests/traces/baseline.json")
    candidate = load_trace("tests/traces/candidate.json")

    # Run the DAG-LCS comparison engine
    report = compare(baseline, candidate)

    # Expressive assertions — raises AssertionError with full report on failure
    assert_no_regressions(
        report,
        max_divergence=0.25,        # TDI threshold [0.0 - 1.0]
        max_cost_increase_pct=5.0,  # Max LLM cost increase allowed
        allow_loops=False,           # Fail if tool loops detected
        max_wasted_effort=0.10,     # Max WEI (failed/retry steps ratio)
    )