Agent Evaluation and Cost per Task

Measure completed work, not tokens.

Talk to us
Duration
30 days
Runs against
One workflow you already have
You get
A workflow scorecard

What a token price does not tell you

An enterprise does not buy tokens. It buys completed work: a resolved customer issue, a reviewed pull request, a reconciled account, a verified incident response. The model’s token price tells you almost nothing about what that costs.

The gap is the work that does not land. Retries, failed tool calls, abandoned branches and the human minutes spent reviewing output all consume budget, and none of them produce an accepted result. A cheaper model that needs three attempts and a reviewer is not cheaper.

So the question an agent workflow has to answer is not what a thousand tokens cost. It is this: how much does it cost to produce one reliable, accepted result?

Cost per completed task, as a division A division drawn vertically. Above the line, six stacked bands show everything one task consumes: model tokens, retries and failed tool calls, tools and APIs, retrieval, sandbox time, and human review and rework. Below the line, a narrowing funnel runs from requests to attempts to accepted results, with aborted and rejected runs leaving the funnel to the right. Those runs leave the denominator but stay in the numerator, which is why the result is cost per completed task rather than cost per call. The star marks the cost per completed task: the number you keep. Everything one task consumes Only the results that are accepted = cost per completed task Model tokens Retries and failed calls Tools and APIs Retrieval Sandbox time Review and rework divided by Requests Attempts Accepted aborted rejected Cost per completed task, as a division A division drawn vertically. Above the line, six stacked bands show everything one task consumes: model tokens, retries and failed tool calls, tools and APIs, retrieval, sandbox time, and human review and rework. Below the line, a narrowing funnel runs from requests to attempts to accepted results, with aborted and rejected runs leaving the funnel to the right. Those runs leave the denominator but stay in the numerator, which is why the result is cost per completed task rather than cost per call. The star marks the cost per completed task: the number you keep. Everything one task consumes Only the results that are accepted = cost per completed task Model tokens Retries and failed calls Tools and APIs Retrieval Sandbox time Review and rework divided by Requests Attempts Accepted aborted rejected
The two halves count different things. Every attempt is priced; only the accepted results divide it. Runs that leave the funnel on the right stay in the numerator, which is where the real number comes from.

What the engagement covers

We instrument your agent workflows so that the answer is evidence rather than an estimate.

  • A task contract An explicit definition of what counts as done, so acceptance is measured rather than asserted.
  • A fixed task cohort A stable set of representative tasks, so numbers can be compared across models and across months.
  • Every attempt counted Retries, failed tool calls, abandoned branches and rework all priced into the task, not excluded from it.
  • Structured traces Per-run records of tools, models, tokens, latency, human time and outcome, attached to one task.
  • Acceptance scored separately from quality So a cheaper model that passes the gate is visible as the better commercial choice.
  • Two figures, not one Marginal cost for comparing one implementation against another, and fully loaded cost for deciding whether to fund the workflow at all.

How the engagement runs

Four weeks, each ending in something you keep. These are the stages we published in the thirty-day rollout, run against your workflow rather than a worked example.

  1. Week 1 Choose one workflow A bounded process with a clear owner, repeated volume and an acceptance test. We write the task contract and define what counts as a failed, an escalated and a completed run. You get: a task contract naming the acceptance test, the budget and the escalation path.
  2. Week 2 Instrument the path Model calls, tool calls, retries, sandbox time, approvals, human minutes, artifacts and acceptance outcomes captured under one task identifier. A small event schema, with the cost fields explicit. You get: an event schema with explicit cost fields, running in your own telemetry.
  3. Week 3 Run a controlled comparison Two models, routing policies or harness configurations on the same task cohort, with tools, data, risk policy and acceptance tests held constant. In shadow mode or behind an approval gate where the workflow has side effects. You get: a like-for-like result table, and the failure tail that explains it.
  4. Week 4 Make one operating decision The system with the best accepted-work economics under your quality and safety gates. That may be a model change, a tool redesign, a shorter context, more automated review — or a decision not to automate this workflow yet. You get: one decision, with the evidence it rests on.

What you leave with

A workflow scorecard rather than a model leaderboard: cost per completed task by cohort, the long tail of failures that drives it, and where architecture changes would move the number.

  • The task contract — yours to reuse on the next workflow.
  • The event schema and traces — in your telemetry, not ours, so the measurement outlives the engagement.
  • The scorecard — marginal and fully loaded cost per accepted task, by cohort.
  • A ranked list of what would move it — each item with the change it needs and what has to come first.

The method, published

We argued this in public before we sold it. The New AI Benchmark Is Cost per Completed Task sets out the metric, a worked comparison in which the cheaper model loses on illustrative assumptions rather than vendor pricing, and what belongs in a serious benchmark. The engagement is that method, run against your system.

Which unit to measure depends on the work: cost per accepted pull request, per verified incident, or per resolved case.

Send us one workflow you already run, and we will tell you what it would take to put a number on it.

Talk to us

Related services

AI and Agent Engineering

Agent Production Readiness

Assess an agent system against the eight layers a production stack needs, and get a ranked plan to close the gaps.

Tell me more
AI and Agent Engineering

AI Governance and Data Boundaries

Model registry, routing policy and a hard data boundary, so regulated workloads can use AI and prove how.

Tell me more
AI and Agent Engineering

LLM Gateway and Model Routing

One vendor-neutral gateway in front of every model, so routing, cost and failover are yours to change.

Tell me more