Failures only show up in production
Teams running voice agents keep saying the same thing: the calls that break are never the ones in the test script. They have to be turned into offline evals by hand.
Open source
Text-only today
Every bad call becomes a test.
CallReplay turns the voice-agent calls that went wrong into regression tests that run in CI before every prompt or model change, and tells you what the change does to quality, cost and latency. Open source, one command, any OpenAI-compatible model.
Baseline vs change
Exit code 1
- regressions (pass -> fail):
- human-handoff
- fixes (fail -> pass):
- cleaning-price
- avg cost/case
- $0.000035 -> $0.000062 (+74.0%)
This change breaks 1 case that passed before and makes each call 74% more expensive.
callreplay demo) with a built-in fake model.The problem
Someone listens to the worst calls every week
One team reviews its 100 worst calls by hand every week. Then a prompt change ships and nobody checks whether those calls still go right.
LLM judges take months to trust
One team took 11 months to get its LLM judge to agree with humans. Meanwhile a "friendlier" prompt can double reply length, and the pass rate does not show it.
How it works
- capture
Point at the bad turn
Load a transcript from Vapi, Retell, LiveKit Agents or plain JSON. Say which agent turn went wrong and what a good reply does.
- replay
Run it against your change
CallReplay sends the same conversation to your new prompt or model, then runs deterministic checks and an LLM judge.
- gate
Fail the PR, show the cost
Compared with the baseline: regressions, fixes, cost per call and latency. Exit code 1 on any regression or on a cost increase you set.
Quickstart
Offline demo, capture, CI
# try it offline, no API key pip install -e . # from the repo: source code coming soon callreplay demo # turn a bad call into a test callreplay show call.json callreplay capture call.json --at-turn 5 \ --tool transfer_to_human --expect "Transfers the caller to a person" # in CI, on every pull request callreplay run --agent agent.json --cases evals \ --baseline .callreplay/baseline.json --fail-on-cost-increase 15 FAILED: 1 regression(s): human-handoff
provisional
Pricing
Open source
$0
The CLI, forever. Apache-2.0.
- Capture, replay, judge, compare
- Markdown and HTML reports
- GitHub Actions example
Team
$99/mo
Hosted dashboard for one team.
- Run history and cost/quality trends
- Judge vs human agreement per rubric
- PR comments
Audio (roadmap)
$499/mo
Evals on the audio, not just the transcript.
- Replay caller audio through your stack
- Interruptions and turn-taking
- Perceived latency
Prices are provisional launch prices and may change.
Get the hosted version first
We are looking for 2–3 teams with voice agents in production to try it on their own calls.
We only use your email to contact you about this. Privacy.