FinOps Charts for Azure AI Agents: Counting Each Run Once
An agent that fails one call in five costs tokens and returns nothing for them. ZopNight’s AI Spend now displays that directly. Open an agent in the Agents view and its drawer carries four charts for the last 30 days: completed and failed runs, tokens and tool calls, and error rate, with a daily table underneath. Each tool lists its name and purpose, and calls are grouped by tool type.
On top of the charts sits one flag. An active agent that fails at least 20 percent of its requests, across at least 20 requests, appears as Failing under Needs attention, pointing at its deployment’s rate limit. No new permission is needed to see any of it.
A Total and Its Parts Were Being Added Together
The charts only mean something if the counts are right, and one count was not. Azure Monitor reports an agent’s responses with a started value that already equals completed plus failed for the same calls. ZopNight fetched that metric unsplit and summed it, which counted every call twice. On the test sandbox, 186 responses were shown for 93 calls.
The error was invisible in shape. A doubled series still rises and falls with the real one, so a chart built on it looks plausible. It only showed up when the displayed number was compared with the number of calls the sandbox had actually made.
One Split Call Per Project
The aggregator now asks Azure Monitor for the metric split by agent and by response status, in a single call per project. It stores three series instead of one: requests, completed responses, and failed responses. Tool calls are split by tool name, so the drawer can show calls per tool type instead of a single total.
Storing completed and failed separately is what makes the error rate possible. A rate needs a numerator and a denominator that are each counted once, and the old summed series could provide neither.
Failing Means 20 Percent of at Least 20
The Failing flag has two conditions, and both matter. The failure share must be 20 percent or more, and the agent must have made at least 20 requests. Without the minimum, an agent that made two calls and lost one would be flagged at 50 percent, which says nothing about the agent.
| Requests | Failed | Flagged |
|---|---|---|
| 10 | 5 | No, below 20 requests |
| 20 | 3 | No, 15 percent |
| 20 | 4 | Yes, 20 percent |
| 100 | 25 | Yes, 25 percent |
This works when the agent’s failures come from a limit on its deployment, which is why the flag points at the rate limit. It is a pointer, not a diagnosis: a failing agent can also fail for reasons that have nothing to do with a limit, and the flag only tells you where to look first.
What Is Kept About a Tool, and What Is Not
For each tool, the discoverer keeps the type, the name and the description. It never stores the tool’s schema, its server URL, or the agent’s own instructions. Those are the parts of an agent definition most likely to contain something sensitive, and none of them is needed to chart how often a tool is called. A monitoring view that never stores them cannot expose them.
Backfilling Without Mixing Old and New
A series still stored under the old summed key is treated as cold once, so the new keys backfill the last 60 days from Azure. Mixing the old and new series would have drawn a chart that halves partway across. Expect one cold metrics sync per agent after deploy.
The corrected counts were reconciled 238 of 238 against Azure Monitor and Cost Management on the sandbox, meaning every figure checked matched its source. That sandbox is a test environment, not customer data, but it is the same comparison that exposed the doubling in the first place.
