How the engine evaluates agent trajectories in single-digit milliseconds.
A technical deep-dive into the deterministic graph algorithms powering AgentDiff: topological trace normalization, modified LCS alignment, k-gram loop detection, and rule-based root cause isolation in CI.
Structural Equivalence Signatures.
Before comparison, raw telemetry traces from LangGraph, CrewAI, OpenAI Agents, and OpenTelemetry are normalized into strongly typed execution DAGs. Every node computes a deterministic structural signature.
Raw Telemetry Trace
Ingests heterogeneous spans and execution logs containing variable timestamps, ephemeral UUIDs, and tool payloads.
Semantic Key Extraction
Drops non-deterministic noise (timestamps, tokens, UUIDs) and sorts input dictionary keys to ensure deterministic ordering.
Canonical Signature Token
Generates an immutable node tuple token ready for high-speed topological longest common subsequence alignment.
Deterministic Hashing Contract
When strict_tool_signatures = true is enabled, AgentDiff performs recursive value hashing on payload contents while applying user-defined regex exclusion masks in agentdiff.toml.
{
"step_id": "step_89f02c",
"parent_id": "step_14a81b",
"step_index": 3,
"step_type": "tool",
"name": "search_vector_database",
"input_payload": { "query": "customer ARR", "top_k": 5 },
"output_payload": { "matched_chunks": 3 },
"metrics": {
"latency_ms": 218.4,
"input_tokens": 420,
"output_tokens": 180,
"cost_usd": 0.0024
}
}Topological Longest Common Subsequence.
Standard string diff algorithms fail on AI agent executions because tool calls contain causal dependencies and cyclic retries. AgentDiff aligns complex execution DAGs through a high-speed 3-phase graph engine.
Topological Dependency Linearization
Linearizes the execution graph via Kahn's algorithm while preserving strict causal dependency edges defined by parent pointers.
Dynamic Programming Table
Constructs an optimized dynamic programming matrix scoring exact signature matches, structural insertions, and tool modifications.
Optimal Path Backtracking
Backtracks the optimal alignment path to mark each step as Matched, Added, Removed, or Modified with sub-cent token delta tracking.
Candidate vs Baseline Execution Sequence
Step-by-step alignment path showing exact match points, syntax modifications, and stagnant loop injections.
| Index | Baseline Step | Candidate Step | Alignment Verdict | Observed Delta |
|---|---|---|---|---|
| 01 | planner:intent | planner:intent | Matched | Identical signature |
| 02 | search_db(query) | search_db(query) | Matched | Identical signature |
| 03 | synthesize_sql | synthesize_sql | Modified | syntax_version: 1 → 2 |
| 04 | — (absent in baseline) | retry_sql_query | Loop Injected | Repetition 1 (+48% tokens) |
| 05 | — (absent in baseline) | retry_sql_query | Loop Injected | Repetition 2 (+100% tokens) |
| 06 | execute_db_pool | execute_db_pool | Matched | Fallback recovery |
Four deterministic metrics. Zero black-box judges.
Every verdict in AgentDiff is mathematically calculated from graph topology and execution telemetry. No subjective prompts or nondeterministic LLM evaluation latencies.
Trajectory Divergence Index
Measures how far the candidate execution drifted from the golden baseline sequence. Computes the ratio of common valid tool steps to total graph size.
Calculated via Kahn's topological sort + DP longest common subsequence in < 2ms.
Stagnant Loop Detection
Identifies agents trapped in infinite retry cycles with stagnant parameter signatures that fail to make state progress.
Distinguishes valid progressive pagination from non-productive retry spirals.
Wasted Effort Index
Calculates the exact percentage of compute time, steps, and token spend allocated to non-productive execution branches.
Quantifies whether prompt updates actually streamline execution or silently bloat costs.
Recovery Step Ratio
Evaluates how quickly your agent recovers from intermediate tool errors compared to the golden baseline.
Ensures prompt refactors don't quietly degrade error-handling resilience.
Know the exact culprit step in plain English.
When a merge gate fails, developers shouldn't spend hours parsing massive JSON telemetry traces. AgentDiff runs a deterministic rules cascade that isolates the culpable tool step and failure mechanism automatically.
Loop Attribution
Identifies the exact cycle length, repeating function name, and stagnant parameter payload that caused the failure.
Fork Point Discovery
Pins the exact step index where the candidate execution graph diverged from the golden baseline DAG.
Resource Attribution
Calculates exact token delta percentages and sub-cent USD cost spikes directly attributed to the culprit node.
Instant root-cause isolation in your terminal and PR comments.
No guessing whether a regression was caused by prompt drift, tool schema changes, or database timeouts. AgentDiff renders the entire execution hierarchy with actionable blame lines.
Declarative governance in agentdiff.toml.
Thresholds and masking rules live directly in your Git repository. Every engineer, branch, and CI runner shares the exact same single source of truth.
Tracked in Git
Gate rules sit right next to your agent code. Changes to tolerance thresholds require PR approval and code review.
Semantic Regex Masks
Automatically strip non-deterministic tokens, ephemeral session IDs, and UUIDs to prevent false-positive CI failures.
Deterministic Exit Codes
Emits clean 0/1 exit codes that instantly integrate into GitHub Actions, GitLab CI, and custom test runners.
Complete TOML Schema Specification
Drop this file at your repository root to configure assertion thresholds, semantic masks, and governance policies.
# agentdiff.toml — Committed repository configuration
[compare]
detect_loops = true
strict_tool_signatures = false
[assertions]
max_divergence = 0.25 # Fails CI if TDI > 0.25
max_wasted_effort = 0.10 # Fails CI if > 10% compute is wasted
allow_loops = false # Fails CI on any tool repetition
max_cost_increase_pct = 5.0 # Fails CI if cost delta > +5%
max_recovery_step_ratio = 1.5 # Fails CI if error recovery degrades > 1.5x
[masking]
ignore_keys = ["timestamp", "session_id", "trace_id", "auth_token"]
regex_patterns = ["^uuid_[0-9a-f-]+$", "^Bearer\\s+.*$"]
[governance]
warn_stale_baseline_days = 30 # Warns when golden baseline exceeds 30 days
flag_threshold_changes = true # Flags PRs that lower gate rigorReady to catch regressions before they hit production?
Install the Python package, record your first golden baseline, and guard your CI pipeline in under 5 minutes.