OpenAI published a piece on managing AI investments in the agentic era and the metric they anchor on is useful work per dollar. Not tokens per second. Not model benchmarks. Useful work per dollar for AI agents, measured against the actual business outcome the agent is supposed to produce. That framing maps cleanly onto Power Platform agents if you wire up the telemetry properly. This walkthrough shows how to instrument a Copilot Studio agent so you can see cost, task success rate, and useful work per dollar per workflow, then use that to decide what to scale and what to kill.
The working result: a Dataverse table plus a Power BI page that gives you a per-agent, per-workflow view of cost efficiency in production. No guessing, no vibes.
Step 1: Define what counts as useful work for the agent
This is the step everyone skips and it is the one that decides whether the whole exercise is worth anything. Useful work is not “the agent responded.” It is the outcome the workflow is supposed to produce.
For a support triage agent, useful work might be “ticket classified correctly and routed without human correction within 24 hours.” For an invoice extraction agent, it is “line items extracted with zero manual edits before posting.” For an internal knowledge agent, it is “user did not open a follow-up ticket on the same topic within 48 hours.”
Write this down as a single boolean per invocation: isUsefulWork = true/false. If you cannot define it in one sentence, the agent probably does not have a clear job yet.
Step 2: Log invocations and outcomes to Dataverse
Create a Dataverse table called AgentInvocation with these columns at minimum: InvocationId, AgentName, WorkflowName, UserId, StartTime, EndTime, InputTokens, OutputTokens, ToolCallsCount, Outcome (choice: Success/Failure/Escalated), IsUsefulWork (bool), and Notes.
In Copilot Studio, add a Power Automate flow at the end of the agent topic that writes a row to this table on every invocation. Pass the run context so you can correlate it later. For the IsUsefulWork field, you have two options: infer it automatically from a downstream signal (was the ticket reopened, did the invoice post, did the user thumbs-down) or capture it with a lightweight feedback prompt at the end of the conversation. I prefer downstream signals because self-reported feedback is noisy.
Microsoft has a good primer on writing to Dataverse from flows if you have not done this pattern before.
Step 3: Pull token and action costs into the same table
Cost is not just tokens. For a Copilot Studio agent using Power Automate flows, tool calls, and premium connectors, the cost per invocation is a stack: model tokens plus per-message consumption plus any premium connector actions triggered inside tool flows.
Build a simple cost model as a Dataverse calculated column or a Power BI measure:
InvocationCost = (InputTokens * InputRate) + (OutputTokens * OutputRate) + (ToolCallsCount * FlowActionCost) + MessageMeterCost
Get the current rates from your tenant’s Power Platform licensing page and your model provider. The exact rates will drift, so store them in a separate PricingRates table with an effective date rather than hardcoding them into the measure. Trust me on this one. I learned the hard way when connector pricing shifted and every dashboard I had was quietly wrong for a month.
Step 4: Build the useful work per dollar view in Power BI
Connect Power BI to the AgentInvocation table. Create three core measures:
TotalCost = SUM(AgentInvocation[InvocationCost])
UsefulWorkCount = CALCULATE(COUNTROWS(AgentInvocation), AgentInvocation[IsUsefulWork] = TRUE)
UsefulWorkPerDollar = DIVIDE([UsefulWorkCount], [TotalCost])
Build one page with a matrix visual: rows are WorkflowName, columns are the three measures plus task success rate and average invocation cost. Add a line chart showing useful work per dollar trending weekly per workflow. That trend line is the one you will actually look at.
Slice by agent, by workflow, by user segment. The interesting patterns are usually in the segments, not the overall number. One workflow can be printing money while another quietly burns budget on retries.
Step 5: Set a scale or kill threshold and review weekly
Numbers without a decision rule are decoration. Set a threshold before you look at the data so you do not rationalize whatever the chart shows.
My rule of thumb: if a workflow’s useful work per dollar is at least 3x the manual baseline (what the same work would cost in human time), it is a scale candidate. If it is below 1x for four weeks running, it gets killed or redesigned. Between 1x and 3x is the improvement zone: look at where the cost concentrates, usually retries, oversized system prompts, or unnecessary tool calls. If you want to understand what is driving those unnecessary tool calls, How Copilot Studio Agent Tool Selection Actually Works Under the Hood covers the orchestrator scoring pass and how to structure tools so the agent stops reaching for the wrong one.
Review weekly. Monthly is too slow, daily is noise.
Final state and one pitfall
You end up with a Dataverse table capturing every invocation, a cost model that reflects real pricing, and a Power BI page that shows useful work per dollar per workflow with a scale-or-kill threshold. That is the whole loop.
The common pitfall: measuring useful work by proxies that are easy to log rather than the ones that actually matter. “User did not thumbs-down” is not the same as “the work was correct.” Spend the extra effort to wire up a downstream signal, even if it means waiting 24 hours to mark an invocation useful. Noisy signals produce confident wrong decisions, and this is a place where being confidently wrong is expensive. If you want to see what happens when agents that looked fine in testing start failing on real signals, Why Do AI Agents Fail in Production When They Worked Fine in Testing walks through exactly that. For more on how I think about this stuff, see my LinkedIn.
Frequently Asked Questions
How do I measure useful work per dollar for AI agents in Power Platform?
Start by defining a clear, binary outcome for each agent workflow, such as whether a ticket was routed correctly or an invoice posted without edits. Then log each invocation to Dataverse with cost and outcome data, and use Power BI to calculate the ratio of successful outcomes to total spend. This gives you a per-agent, per-workflow view of cost efficiency rather than relying on vague performance indicators.
What is useful work in the context of a Copilot Studio agent?
Useful work refers to the specific business outcome an agent is designed to produce, not simply the fact that it generated a response. For example, a support triage agent produces useful work only if the ticket is classified and routed correctly without human correction. If you cannot define that outcome in a single sentence, the agent likely lacks a clear purpose.
How do I track AI agent invocation costs in Dataverse?
Create a Dataverse table that records key details for every agent run, including token counts, tool calls, timestamps, and the workflow outcome. A Power Automate flow triggered at the end of each Copilot Studio topic can write this data automatically. Over time, this log gives you the raw numbers needed to calculate cost per successful outcome.
When should I use downstream signals instead of user feedback to measure agent success?
Downstream signals, such as whether a ticket was reopened or an invoice posted without edits, are generally more reliable than asking users to rate the interaction directly. Self-reported feedback tends to be inconsistent and easy to ignore, while automated signals reflect actual business outcomes. Use feedback prompts only when no suitable downstream signal exists for the workflow.
This post was inspired by How to manage AI investments in the agentic era via OpenAI.
Leave a Reply