Skip to content
AI Agents

Agent Trajectory Evaluator

Replay agent runs, score each step and find where agents go off the rails.

Advanced26h1–3 people15/20 fit
TL;DR

Build Agent Trajectory Evaluator for teams building ai agents in ~26h with 1–3 people. Stack: OpenTelemetry + LLM + Next.js + Postgres.

Why it wins: Agent reliability is the top problem for AI teams right now.

Build flow

01 · Problem
Agent failures are hard to debug because the bad decision happened ten steps earlier.
02 · Solution
Log agent traces, replay them as a timeline, score each step with rubric-based LLM judges and cluster failures by root cause.
03 · Build
  1. 1Capture traces with OpenTelemetry
  2. 2Store spans in Postgres
  3. 3Score steps with an LLM judge
  4. 4Cluster failure explanations
04 · Demo
Load 20 runs and show the cluster of failures caused by one bad tool description.

MVP checklist

  • Trace timeline
  • Per-step scoring
  • Failure clustering
  • Run comparison

Tech stack

TechRole
OpenTelemetryTraces + metrics
LLMLanguage model
Next.jsFrontend + API routes
PostgresRelational database

Reference repos

Open-source projects you can build on. Links checked 2026-10-31.

Related ideas

More AI Agents

  • AI Agents

    Agent Run Budget Guard

    Stop runaway agents by capping tokens, cost and steps per task

    Intermediate12h1–2
    Node.jsVercel AI SDKSQLite
    17/20

  • AI Agents

    Accessibility Fix PR Agent

    An agent finds accessibility issues and opens a PR that fixes them

    Advanced26h1–3
    axe-corePlaywrightLLM
    16/20

  • AI Agents

    Dependency Upgrade Agent

    An agent upgrades a dependency, fixes breaking changes and opens a PR with test results

    Advanced28h1–3
    LLMGitHub APINode.js
    15/20