Back to Projects

DiffSmith

In progress

An agentic AI pull-request reviewer I'm building. It gives an LLM real tools—reading files, searching the codebase, checking git history—instead of a single prompt, with an eval harness to keep its reviews honest.

May 1, 2026
TypeScriptNest.jsAnthropic SDKPostgreSQLPrisma

What I'm Building

DiffSmith is a side project I'm actively building: a tool that reviews pull requests with an LLM agent that actually investigates the code, rather than skimming a diff and guessing.

The Idea

Most "AI review" tools paste a diff into a single prompt and hope for the best. The trouble is that a diff on its own is missing context, like the rest of the function being changed, where it's called from, and why it looked the way it did. DiffSmith gives the model tools instead. It can open files, search the codebase with ripgrep, and check git blame and history, taking several turns to gather what it needs before it says anything. The goal is a review that reads like a careful teammate wrote it, not a linter with a thesaurus.

The Differentiator: Evals

The interesting part isn't the prompt, it's knowing whether a change to the prompt made the reviews better or worse. So DiffSmith is built around an eval harness: a hand-labeled corpus of real pull requests with "good review" ground truth, and a regression runner that scores the agent against it on every change. Without that, you're just vibe-checking an LLM and trusting your gut. With it, you can actually iterate on evidence.

How It's Built

  • TypeScript end to end, the stack I work in every day.
  • A Commander.js CLI (diffsmith review <diff>) as the first surface.
  • The Anthropic SDK for Claude, leaning on tool calling, structured output, and prompt caching.
  • A Nest.js service with a GitHub webhook receiver, so reviews can run automatically on real pull requests.
  • Postgres via Prisma for persisting reviews, runs, and eval results, with JSONB for the review payloads.
  • Vitest with testcontainers for integration tests against a real Postgres rather than a mock.

Why

I've spent most of my career deep in payments. DiffSmith is me deliberately working in a different corner, agentic systems and developer tooling, and building it carefully enough that the design and the trade-offs hold up to scrutiny. Treating the evals as the core, instead of an afterthought, is the part I think most "ship an AI feature" efforts skip.

Status

In active development. The CLI and the Anthropic integration are working; next up is the tool-calling agent and the eval corpus, then the Nest.js service and GitHub integration, and finally a write-up on what I learned. Source and that write-up to follow.