How to Measure the Cost and Output of AI Coding Agents | Pensero Blog[Skip to content](#main) [Pensero](/) - Platform Measure - [Next movesWhat to improve, why, and what to do next](/platform/next-moves) - [Agentic ROICost per unit of delivered work](/platform/agentic-roi) - [AI DeploymentAdoption, output and quality of AI](/platform/ai-intelligence) - [Delivery IntelligenceOutput and flow by source](/platform/delivery-intelligence) Govern - [Agent Traffic ControlUnderstand agent usage and choose models by delivery](/platform/agent-traffic-control) - [BenchmarkingCompare teams, agents and models](/platform/talent-benchmarking) - [Code QualityDefects and rework by source](/platform/code-quality) - [CapEx & FinanceCapitalization and spend allocation](/platform/capex-analysis) [Pensero MCPNewYour next moves in any assistant](/platform/mcp) - Use cases By role - [CEOs](/use-cases/roles-ceo) - [CTOs](/use-cases/roles-cto) - [CFOs](/use-cases/roles-cfo) - [Product leaders](/use-cases/roles-product-leaders) - [Engineering managers](/use-cases/roles-engineering-managers) - [Investors](/use-cases/roles-investors) Compare Pensero - [DX](/use-cases/competitors-dx) - [Swarmia](/use-cases/competitors-swarmia) - [Haystack](/use-cases/competitors-haystack) - [Span](/use-cases/competitors-span) - [Jellyfish](/use-cases/competitors-jellyfish) - [Waydev](/use-cases/competitors-waydev) - [LinearB](/use-cases/competitors-linearb) - [Faros AI](/use-cases/competitors-farosai) - [Larridin](/use-cases/competitors-larridin) - [Weave](/use-cases/competitors-weave) - [Exceeds AI](/use-cases/competitors-exceeds-ai) - [How it works](/how-it-works) - [Customers](/customers) - [Pricing](/#pricing) - Resources - [About](/about) - [BlogNews, tips, and dev culture](/blog) - [Help CenterGuides for setup, integrations and security](https://help.pensero.ai) [Log in](/auth/login/)[Book a scoping session](/book-demo) [Log in](/auth/login/)[Book a scoping session](/book-demo) [Blog](/blog) / Other resources # How to measure the cost and output of AI coding agents Measure what Devin, Claude Code and Codex deliver and what each unit of work costs. A method in delivery points, the five levels and an FAQ. ![](/assets/WF5wXySb5oYBsfMDPbU0hRuXY3M.LvpWanuL_ZHEg5Y.webp) ![](/assets/GjPJ8lgQ2s9KH4YirhymwwZxVY.WQPn6KWn_ZcotNb.webp)Pensero·Pensero Marketing·Nov 10, 2026 Short answer To measure an AI coding agent, measure the finished work it delivered, not the sessions or tokens it used. Score each pull request and ticket in delivery points, add its cost and its quality, and compare the cost per delivered point with the same number for people and for other agents. Pensero does this in one unit for people, vendors and agents. ## Why agent spend is hard to read Agents such as Devin, Claude Code and Codex now open pull requests, close tickets and run for hours without a person at the keyboard. The bill comes as seats, model calls and agent runs. None of these tell you what the agent delivered. Tokens and sessions measure effort. A long session can produce a small fix. A short session can produce a large change. If you count tokens, you reward the agent that uses the most, not the agent that delivers the most. ## Why is cost per merged PR not enough? Some teams divide the agent cost by the number of merged pull requests. This is better than counting tokens, but a merged pull request can be a one-line fix or a change across many systems. An agent that merges many small fixes looks cheaper than an agent that merges a few hard changes. Divide the cost by delivery points, so that each pull request counts for its size and difficulty. ## Step 1: Score the work, not the activity Score every finished pull request and ticket by size and difficulty. In Pensero, delivery points multiply the magnitude of each pull request by its complexity. The score is the same for a person, an AI-assisted engineer and an agent, so you compare work, not tools. ## Step 2: Attach the cost to each unit of work Add what the work cost: model spend, agent runs and seats for the agent, and people's time for the work that people did. Then divide by the delivery points. The result is the cost per delivered point, for each owner, team, tool and model. ## Step 3: Check that quality held Cheaper work is not better work if it comes back. Read rework, reverts and defects next to the points. An agent that delivers many points with a lot of rework costs more than its bill shows. ## Step 4: Compare agents with people and with each other Put the cost per delivered point side by side: people, AI-assisted engineers and each agent. Compare by type of work, because an agent can do well on small fixes and badly on cross-system changes. ## Step 5: Act on the result Keep the expensive model where it delivers. Move the other work to a lower-cost model. Give the agent more of the work where it delivers at quality, and less where it does not. Then measure again. ## The five levels of AI in engineering The five levels describe who does the work: human-only, AI-assisted, AI reviews, orchestrated agents and fully autonomous agents. Most organizations run several levels at once, and the highest level is not always the right one. Measure cost per delivered point at each level before you move more work up. ## How Pensero measures agents The delivery point is one unit of finished work, scored by size and difficulty, the same way for people, vendors and agents. Pensero reads it next to cost and quality, and traces it to the pull request and ticket behind it. - Pensero tracks what Devin, Claude Code and Codex deliver, at what cost and with what quality. - With [Agent Traffic Control](/platform/agent-traffic-control), Pensero measures the cost per delivered point for each owner. It keeps premium models where they deliver and routes the others to lower-cost models. It runs on-prem, on your host or in your VPC. - See the [integrations](/integrations) and [current pricing](/#pricing), or [book a scoping session](/consulting). ## Frequently asked questions How do you measure the output of an AI coding agent? Score the finished pull requests and tickets that the agent delivered by size and difficulty. Do not count sessions, prompts or tokens, because they measure effort, not delivered work. What is the best metric for AI agent cost? Cost per delivered point: what each unit of finished work cost, with quality read next to it. It works for people and agents in the same unit. Is cost per merged PR a good metric for AI agents? Only in part. A merged pull request can be a one-line fix or a change across many systems. Score each pull request by size and difficulty, then read the cost per delivered point. Is token spend a good measure of agent value? No. Token spend tells you what an agent used. It does not tell you what the agent delivered or whether the work held. Can you compare Devin, Claude Code and Codex? Yes, if one platform scores the work of each agent the same way. Then you compare cost per delivered point and quality by type of work. Can you compare agents with human engineers? Yes. Score all finished work in the same unit. Then cost per delivered point shows where an agent is cheaper for the same quality, and where people still do better. How do you know if an agent is creating rework? Read rework, reverts and defects next to the delivery points of the agent. If they rise with its output, part of the output is not finished work. Should every task go to the most capable model? No. Keep the expensive model where it delivers and move the other work to a lower-cost model. Measure cost per delivered point after each change. Can you measure agents without sending code to a vendor cloud? Yes. Pensero Agent Traffic Control runs on-prem, on your host or in your VPC. ## Related posts - [![](/assets/WF5wXySb5oYBsfMDPbU0hRuXY3M.LvpWanuL_ZHEg5Y.webp) ### Devin Alternatives: 9 AI Coding Tools and How to Measure ROI Compare AI coding assistants and autonomous agents, their oversight trade-offs, and ways to evaluate delivery, quality and return on investment. Pensero Pensero Marketing Apr 21, 2026](/blog/devin-alternatives) - [![](/assets/WF5wXySb5oYBsfMDPbU0hRuXY3M.LvpWanuL_ZHEg5Y.webp) ### How to Measure and Improve Product Development Performance Connect customer value, delivery, quality, strategic alignment, and investment efficiency in a balanced product development performance review. Pensero Pensero Marketing Aug 10, 2026](/blog/product-development-performance) - [![](/assets/WF5wXySb5oYBsfMDPbU0hRuXY3M.LvpWanuL_ZHEg5Y.webp) ### AI Engineer vs Software Engineer: Key Differences Compare AI and software engineers by responsibilities, skills, quality standards, and risk, with guidance on how they work together and when to hire each. Pensero Pensero Marketing Aug 10, 2026](/blog/ai-engineer-vs-software-engineer) Next step ## Find your level of AI in engineering See the 5 levels, from human-only to fully autonomous agents, and what moving up is worth. [See the 5 levels →](/how-it-works#levels) Next step ## Scope your audit in 30 minutes We look at your tools, AI agents and vendors and tell you what we would measure first. [Book a scoping session](/book-demo)[Start free](/auth/signup/) © 2026 Product shots use illustrative data. Platform - [Next moves](/platform/next-moves) - [Agentic ROI](/platform/agentic-roi) - [AI Deployment](/platform/ai-intelligence) - [Delivery Intelligence](/platform/delivery-intelligence) - [Benchmarking](/platform/talent-benchmarking) - [Code Quality](/platform/code-quality) - [Agent Traffic Control](/platform/agent-traffic-control)New - [CapEx & Finance](/platform/capex-analysis) - [Pensero MCP](/platform/mcp)New - [Integrations](/integrations) Use cases - [CEOs](/use-cases/roles-ceo) - [CTOs](/use-cases/roles-cto) - [CFOs](/use-cases/roles-cfo) - [Product leaders](/use-cases/roles-product-leaders) - [Engineering managers](/use-cases/roles-engineering-managers) - [Investors](/use-cases/roles-investors) - [Compare Pensero](/use-cases/competitors-dx) Company - [About](/about) - [How it works](/how-it-works) - [Consulting](/consulting) - [Careers](/careers) - [Blog](/blog) - [LinkedIn](https://www.linkedin.com/company/penseroai/) - [Support](/support) - [Help Center](https://help.pensero.ai) - [Other resources](/blog/other-resources) - [Glossary](/glossary) Offices - [Barcelona](https://maps.app.goo.gl/6pfxKiKFNKzLPPnB6) - [Madrid](https://maps.app.goo.gl/jEpYBUR4E6AqZs4R6) - [New York](https://maps.app.goo.gl/szk58Jf2MBq14MgAA) - San FranciscoSoon Legal - [Privacy policy](/privacy-policy) - [Cookie policy](/cookie-policy) - [Terms of service](/terms) - [DPA](/dpa) - [Security](https://pensero.trust.site)