Why agent spend is hard to read
Agents such as Devin, Claude Code and Codex now open pull requests, close tickets and run for hours without a person at the keyboard. The bill comes as seats, model calls and agent runs. None of these tell you what the agent delivered.
Tokens and sessions measure effort. A long session can produce a small fix. A short session can produce a large change. If you count tokens, you reward the agent that uses the most, not the agent that delivers the most.
Why is cost per merged PR not enough?
Some teams divide the agent cost by the number of merged pull requests. This is better than counting tokens, but a merged pull request can be a one-line fix or a change across many systems. An agent that merges many small fixes looks cheaper than an agent that merges a few hard changes. Divide the cost by delivery points, so that each pull request counts for its size and difficulty.
Step 1: Score the work, not the activity
Score every finished pull request and ticket by size and difficulty. In Pensero, delivery points multiply the magnitude of each pull request by its complexity. The score is the same for a person, an AI-assisted engineer and an agent, so you compare work, not tools.
Step 2: Attach the cost to each unit of work
Add what the work cost: model spend, agent runs and seats for the agent, and people's time for the work that people did. Then divide by the delivery points. The result is the cost per delivered point, for each owner, team, tool and model.
Step 3: Check that quality held
Cheaper work is not better work if it comes back. Read rework, reverts and defects next to the points. An agent that delivers many points with a lot of rework costs more than its bill shows.
Step 4: Compare agents with people and with each other
Put the cost per delivered point side by side: people, AI-assisted engineers and each agent. Compare by type of work, because an agent can do well on small fixes and badly on cross-system changes.
Step 5: Act on the result
Keep the expensive model where it delivers. Move the other work to a lower-cost model. Give the agent more of the work where it delivers at quality, and less where it does not. Then measure again.
The five levels of AI in engineering
The five levels describe who does the work: human-only, AI-assisted, AI reviews, orchestrated agents and fully autonomous agents. Most organizations run several levels at once, and the highest level is not always the right one. Measure cost per delivered point at each level before you move more work up.
How Pensero measures agents
The delivery point is one unit of finished work, scored by size and difficulty, the same way for people, vendors and agents. Pensero reads it next to cost and quality, and traces it to the pull request and ticket behind it.
- Pensero tracks what Devin, Claude Code and Codex deliver, at what cost and with what quality.
- With Agent Traffic Control, Pensero measures the cost per delivered point for each owner. It keeps premium models where they deliver and routes the others to lower-cost models. It runs on-prem, on your host or in your VPC.
- See the integrations and current pricing, or book a scoping session.
Frequently asked questions
How do you measure the output of an AI coding agent?
Score the finished pull requests and tickets that the agent delivered by size and difficulty. Do not count sessions, prompts or tokens, because they measure effort, not delivered work.
What is the best metric for AI agent cost?
Cost per delivered point: what each unit of finished work cost, with quality read next to it. It works for people and agents in the same unit.
Is cost per merged PR a good metric for AI agents?
Only in part. A merged pull request can be a one-line fix or a change across many systems. Score each pull request by size and difficulty, then read the cost per delivered point.
Is token spend a good measure of agent value?
No. Token spend tells you what an agent used. It does not tell you what the agent delivered or whether the work held.
Can you compare Devin, Claude Code and Codex?
Yes, if one platform scores the work of each agent the same way. Then you compare cost per delivered point and quality by type of work.
Can you compare agents with human engineers?
Yes. Score all finished work in the same unit. Then cost per delivered point shows where an agent is cheaper for the same quality, and where people still do better.
How do you know if an agent is creating rework?
Read rework, reverts and defects next to the delivery points of the agent. If they rise with its output, part of the output is not finished work.
Should every task go to the most capable model?
No. Keep the expensive model where it delivers and move the other work to a lower-cost model. Measure cost per delivered point after each change.
Can you measure agents without sending code to a vendor cloud?
Yes. Pensero Agent Traffic Control runs on-prem, on your host or in your VPC.