Measure where your team’s work piles up. Put one bought agent on the toil nobody wants. Build only the platform that line needs. Track what merges, and what it costs.
Really useful walkthrough, and I like that it starts from where work actually waits rather than which agent to buy. The one bit I'd push on is cost per merged PR as the headline metric. Merged isn't the same as good: an agent that splits work into lots of tiny PRs looks cheap on that number, and a change that gets reverted or quietly adds maintenance load three months later still counts as a win. I'd want something like cost per merged change that survives 30 days without a revert or follow-up fix sitting next to it, otherwise the factory risks optimising for throughput it'll pay for later.
My checklist says "track cost per outcome, next to revert rate," and that's too thin.
A revert only catches the loud failures. It misses exactly what you describe: work split into tiny PRs to look cheap, and changes that quietly cost maintenance later.
Looking back at my sources, several teams already measure it the way you suggest.
Lovely explanation. Many thanks for the reference list.
Really useful walkthrough, and I like that it starts from where work actually waits rather than which agent to buy. The one bit I'd push on is cost per merged PR as the headline metric. Merged isn't the same as good: an agent that splits work into lots of tiny PRs looks cheap on that number, and a change that gets reverted or quietly adds maintenance load three months later still counts as a win. I'd want something like cost per merged change that survives 30 days without a revert or follow-up fix sitting next to it, otherwise the factory risks optimising for throughput it'll pay for later.
Thank you, you're right.
My checklist says "track cost per outcome, next to revert rate," and that's too thin.
A revert only catches the loud failures. It misses exactly what you describe: work split into tiny PRs to look cheap, and changes that quietly cost maintenance later.
Looking back at my sources, several teams already measure it the way you suggest.
- Swarmia describes companies tracking rework over a six-month window, to see how much of their faster shipping they pay for later. https://www.swarmia.com/blog/productivity-impact-of-ai-coding-tools/
- Anthropic's playbook tracks rework cycles per change. https://claude.com/resources/articles/the-ai-native-sdlc-playbook
- Intercom watched downtime from bad changes next to its deployment count. https://ideas.fin.ai/p/2x-nine-months-later
So the better headline is yours: cost per merged change that survives 30 days without a revert or follow-up fix.
Thanks for taking the time to share your feedback on this.