Benchmark: with and without Quack

Measured on 2026-11-08 with Claude Code 2.1.318. Every number below is a real run; nothing is estimated.

Method

• Project: a copy of a real Next.js site (the quack.build landing, 280 tracked files). Each run starts from a fresh copy.
• Configs: base is Claude Code as installed on the test machine. quack is the same, plus --plugin-dir pointing at this plugin and the code map already built (as if /quack:index had run once).
• Main model: --model opus or --model sonnet, default effort. Subagents use their own model.
• Cost: total_cost_usd and modelUsage from claude -p --output-format json. They include subagents.
• Runs: each task was run twice per config. The install ping was off (QUACK_NO_PING=1).
• The same floor in every run: the test machine loads a large personal setup (global CLAUDE.md, rules, other plugins), about 40k tokens before the first message, in both configs. A lighter setup has a lower floor, so the same differences weigh more in percent.

The script is bench/run.sh and the tasks are in tasks.txt. To repeat a run:

plugins/quack/bench/run.sh <any-git-repo> quack opus t4 ./bench-out

Summary

Every t4 and t6 run finished the job: a shared helper, both routes using it, and a passing node:test file.

What it means

1. On a single short task, Quack doesn't change the bill. The differences are -3% to +4%, which is noise. On these tasks Opus worked alone and called no duck. That was the right call: a subagent re-reads what it needs in a fresh context, and on a short job that costs more than it saves.
2. The main model is the biggest lever. The same multi-file job cost $0.25 on Sonnet and $0.56 on Opus, with the same result.
3. The ducks buy quality and a smaller chat, not a cheaper task. "Pia plans, Kit builds" put Opus on both the thinking and the code, in their own contexts. It cost $0.92 from an Opus chat and $0.63 from a Sonnet chat, against $0.56 for Opus alone. Use them on big or unclear tasks, where a plan and a clean context matter. Don't use them on everyday edits.
4. What a one-prompt test can't show is where Quack is built to save: long chats. That covers the /clear and /compact reminders at 100k tokens, the warning when a pause expired the cache, and the code map that stops whole-file reads over a long session. To measure those, compare /usage over a week with and without Quack.

Limits of this test

• Two runs per task, on one project, on one machine.
• Costs depend on Anthropic's prices on the test day.
• Wall time is not reported. With background subagents, duration_ms covers only the main loop.

Every run

Tasks: t1 = Explain a flow, t2 = FAQ edit (2 languages), t3 = Explore a section, t4 = Rate limiter: helper, 2 routes, test, t5 = Review all routes, fix one, diary, t6 = t4, forced "Pia plans, Kit builds".

The same page, with the raw data, lives in the plugin repo: bench/RESULTS.md.