Show HN: TracePact – Catch tool-call regressions in AI agents before prod

1 point

3 months ago

I kept running into the same problem: I'd change a prompt or update my model, the agent output looked fine, but the tool sequence was completely different. It stopped reading config before deploying. It ran npm run build instead of npm test. The bug showed up days later in prod.

So I built TracePact. Record a known-good run as a cassette, diff against new runs, get a clear report of what changed:

- read_file (seq 0) (removed) ~ bash.cmd: "npm test" -> "npm run build" Summary: 1 removed, 1 arg changed[BLOCK]

It classifies changes as block (structural) or warn (args only), so you can gate CI with --fail-on warn. You can filter noise with --ignore-keys timestamp and --ignore-tools read_file.

No API calls needed for replay/diff. Works with any LLM provider. Vitest integration for writing assertions on tool traces.

https://github.com/dcdeve/tracepact