How do I build an AI agent in Copilot Studio for the first time?

Start by writing a tight Instructions prompt that gives the agent one clear job and explicit boundaries, then attach a knowledge source like SharePoint, and connect a single Power Automate flow as a tool. Test it before publishing to a channel like Teams. Keeping the scope narrow on your first build makes it much easier to identify where things go wrong.

What should I write in the Instructions field when creating a Copilot Studio agent?

Instead of writing a vague description, define exactly what the agent can and cannot do, which sources it should use, and how it should respond when it does not know the answer. Keeping the instruction set under 500 tokens helps maintain reliability, as longer prompts can reduce consistency.

When should I turn on general knowledge in Copilot Studio?

Only enable general knowledge if your agent genuinely needs to draw on information beyond your attached sources. For internal use cases like a policy or FAQ agent, leaving it off produces more trustworthy answers because responses are grounded in your own documents and can be cited.

Why does my Copilot Studio agent give incomplete answers from PDF files?

The knowledge indexer in Copilot Studio handles standard PDF and Word files well but struggles with scanned images embedded inside PDFs. If your documents contain scanned pages rather than real text, the agent may only read the portions it can process and skip the rest without flagging it.

How do I write good descriptions for Copilot Studio agent tools?

Write descriptions using the vocabulary your users actually type, not technical or architectural language. Include phrases like 'use when the user asks about...' so the orchestrator can match the tool to real user intents. Describing when to use a tool, rather than how it works internally, consistently improves routing accuracy.

Why does my Copilot Studio agent keep routing to the wrong tool?

Tool selection is driven by a relevance score calculated at runtime, not a manual lookup, so vague or technical definitions cause silent misrouting. If your tool names and descriptions are written for developers rather than the planner, the orchestrator cannot reliably match them to user input. Rewriting names as verb phrases and descriptions in plain user language usually resolves the problem.

What is the best way to name tools in a Copilot Studio agent?

Name tools as verb phrases that describe a clear action, such as GetOpenInvoicesForCustomer rather than SAPQuery. The name is scored alongside the description, so it carries relevance signal and should reflect user intent rather than the underlying system or API.

How do I define parameters so a Copilot Studio agent can fill them correctly?

Give each parameter a descriptive name and a clear explanation of what value it expects and where that value comes from in the conversation. A parameter named 'id' with no context forces the planner to guess, while a name like 'customerName' with a description reduces binding errors and makes extraction from user utterances more reliable.

What is the most expensive part of building enterprise AI agents?

The biggest costs in enterprise AI agents are not the model itself but the surrounding layers: data grounding, governance policies, tool configuration, and measurement pipelines. These are bespoke to your organisation and cannot be swapped out the way a model can, which is why they consume the most engineering time and budget.

How do I choose the right AI model for an enterprise agent?

Model selection matters far less than most teams assume, since modern platforms let you swap models with minimal effort. A better approach is to spend the majority of your planning budget on data grounding, connector policies, and measurement before you finalise any model choice.

Why does my enterprise AI agent give unreliable answers despite a strong model?

Unreliable outputs are usually a grounding problem, not a model problem. If the agent lacks access to your organisation's policies, naming conventions, and internal context, even a frontier model will produce generic or incorrect responses. Moving that knowledge out of long system prompts and into structured, versioned records tends to improve consistency.

When should I move AI policy instructions out of the system prompt?

Once your system prompt grows beyond a few hundred tokens, reliability starts to drop and maintenance becomes difficult. Migrating policy instructions into a managed data layer gives each rule a clear owner and version history, and it keeps the prompt focused on behaviour rather than organisational context.

How does MCP work under the hood when an agent calls a tool?

MCP is a stateful client-server protocol built on JSON-RPC 2.0, not a simple function-calling wrapper. Before any tool call happens, the client and server go through a capability negotiation handshake, the server declares what it supports, and only then does tool discovery take place. The connection stays live as a session rather than being a stateless request like a REST call.

What is the MCP handshake and why does it matter?

When an MCP client connects to a server, it sends an initialize request containing its protocol version and capabilities, and the server responds with its own. The client then sends an initialized notification to confirm the exchange before any tools are listed or called. Skipping this step or misunderstanding it is a common reason real implementations break outside of demos.

What is the difference between MCP and a standard REST connector?

A REST connector is stateless, meaning each request stands alone with no shared context. MCP maintains a live session, negotiates capabilities upfront, and exposes tools, resources, and prompts as distinct primitives. This makes it better suited for agents that need to understand what a server can do before deciding how to use it.

Why does MCP use JSON-RPC 2.0 instead of a simpler format?

JSON-RPC 2.0 gives MCP a lightweight but structured message format that distinguishes requests from notifications and pairs responses to the correct request using an id field. This structure supports the sequenced, session-based communication that MCP requires, while remaining simple enough to implement across different clients and servers.

How do I measure useful work per dollar for AI agents in Power Platform?

Start by defining a clear, binary outcome for each agent workflow, such as whether a ticket was routed correctly or an invoice posted without edits. Then log each invocation to Dataverse with cost and outcome data, and use Power BI to calculate the ratio of successful outcomes to total spend. This gives you a per-agent, per-workflow view of cost efficiency rather than relying on vague performance indicators.

What is useful work in the context of a Copilot Studio agent?

Useful work refers to the specific business outcome an agent is designed to produce, not simply the fact that it generated a response. For example, a support triage agent produces useful work only if the ticket is classified and routed correctly without human correction. If you cannot define that outcome in a single sentence, the agent likely lacks a clear purpose.

How do I track AI agent invocation costs in Dataverse?

Create a Dataverse table that records key details for every agent run, including token counts, tool calls, timestamps, and the workflow outcome. A Power Automate flow triggered at the end of each Copilot Studio topic can write this data automatically. Over time, this log gives you the raw numbers needed to calculate cost per successful outcome.

When should I use downstream signals instead of user feedback to measure agent success?

Downstream signals, such as whether a ticket was reopened or an invoice posted without edits, are generally more reliable than asking users to rate the interaction directly. Self-reported feedback tends to be inconsistent and easy to ignore, while automated signals reflect actual business outcomes. Use feedback prompts only when no suitable downstream signal exists for the workflow.

Why do AI agents fail in production after passing all their tests?

AI agents often fail in production because test inputs are too clean and predictable compared to what real users actually type. Other common causes include tools that return vague responses the agent cannot reason about, and a lack of decision logging that makes it nearly impossible to diagnose what went wrong.

How do I stop my AI agent from choosing the wrong tool for a user request?

Write tool descriptions as retrieval hints rather than documentation, and use specific parameter names that clearly signal their purpose. Ambiguous phrasing from real users causes the agent to score tools incorrectly, so tighter tool contracts reduce misrouting significantly.

What is silent failure in an AI agent and why does it matter?

Silent failure happens when an action inside an agent workflow fails but the agent still generates a confident-sounding response, giving the user no indication anything went wrong. This is dangerous because the underlying task was never completed, and the problem may not surface until days or weeks later.

When should I add decision logging to an AI agent?

Decision logging should be built in before the agent goes live, not added after something breaks. Without it, you can see that the agent ran but not why it chose a particular tool or produced a specific response, which makes debugging little more than guesswork.

How does Copilot Studio agent tool selection actually work at runtime?

Rather than simply reading a list of tools and picking the best match, the orchestrator runs a planning pass that scores each tool based on relevance to the current turn. It factors in the tool name, description, input parameter names, and any enum values before selecting a candidate and resolving the required inputs. This scoring process is closer to retrieval than traditional routing.

Why does my Copilot Studio agent keep picking the wrong tool?

Vague tool names and thin descriptions are the most common cause, as the orchestrator scores tools for relevance and poorly described tools rank badly even if they are logically the right choice. Writing descriptions as retrieval hints rather than documentation, and being specific with parameter names, will improve selection accuracy significantly.

How do I write better tool descriptions for a Copilot Studio agent?

Instead of writing descriptions like reference documentation, treat them as signals that help the orchestrator match the tool to user intent. Be specific about what the tool does, what inputs it expects, and when it should be used rather than a similar tool. Precise parameter names and enum values also feed into the scoring process.

When should I split one Copilot Studio agent into multiple agents instead of adding more tools?

As the number of attached tools grows, the scoring pass has more candidates to evaluate and the risk of the wrong tool being selected increases. If you find yourself adding a large number of tools to a single agent, splitting responsibilities across multiple agents can improve reliability and reduce latency caused by retry loops.

Tag: Copilot Studio

How to Build Your First AI Agent in Copilot Studio

Microsoft published a guide on how to build an AI agent aimed at anyone, and the framework is genuinely sound: define the job, give the agent knowledge, give it tools, test it, let it run. What the article does not give you is the clicks. So this is how to build an AI agent in Copilot Studio following that exact recipe, end to end. By the end you have a working internal FAQ agent that can also create a ticket, published to Teams. I built a variant of this internally last quarter, and the framework holds up. The gaps between the steps are where it gets interesting.

Step 1: Define One Job Before You Build an AI Agent in Copilot Studio

Microsoft’s guide says start with a clear purpose. In practice this means the Instructions field, and this is where most first builds go wrong. People write vibes: “You are a helpful assistant for IT questions.” That constrains nothing. Write scope enforcement instead:

You answer questions about internal IT policies using only the attached knowledge sources. If the answer is not in the knowledge sources, say so and offer to create a support ticket. Do not answer questions about HR, payroll, or anything outside IT. Do not guess at policy details.

Keep it under 500 tokens. I covered why in the business skills post: instruction sets that grow past a few hundred tokens start degrading reliability. One job. Tight boundaries. Explicit refusal behavior.

Step 2: Attach a Knowledge Source the Agent Can Cite

In the agent’s Knowledge section, add your SharePoint site. Use the site URL, not a document library deep link, and check that the files are PDF or Word. The indexer handles those well. It handles scanned images inside PDFs poorly, and you will not find out until the agent confidently answers from the two readable pages and ignores the rest.

Turn on “Use general knowledge” only if you have a reason. For an internal policy agent, leave it off. Grounded-only answers with citations beat fluent answers with no source. If you are choosing between SharePoint and Dataverse here, I compared them in a separate post. For a first build, SharePoint is the shorter path. If you plan to give the agent its own identity and audit trail inside Dataverse, Entra Agent ID gives every Dataverse agent its own identity and is worth reading before you commit to the data layer.

Step 3: Give It One Tool With a Power Automate Flow

Microsoft’s guide calls this “giving the agent the ability to act.” Concretely: build a flow with the When an agent calls the flow trigger, add two text inputs (ticketTitle, ticketDescription), create the ticket row wherever your ticketing lives, and end with Respond to the agent returning the ticket number and status. Do not return a bare “success” flag. Return state the agent can reason about on the next turn. The Copilot Studio docs cover the wiring, but they undersell the description field.

Then add the flow as a tool on the agent, and write the tool description as a retrieval hint, not documentation: Creates an IT support ticket when the user reports a problem that the knowledge sources cannot resolve. The orchestrator scores this against user input to decide when to call it. I went deep on this in How to Design Tool Definitions for Copilot Studio Agents. One tool for now. Resist adding more until the first one behaves.

Step 4: Test in the Canvas With the Activity Map Open

Open the test pane and turn on the activity map. Now test the way real users will actually talk, not the way you wish they would. Three categories: questions the knowledge sources answer, requests that should trigger the ticket flow, and deliberately bad inputs. “my thing is broken.” “can you help.” A question about payroll that should get refused.

The activity map shows you which knowledge source or tool the orchestrator picked and why. When it picks wrong, the fix is almost always the tool description or the instructions, not the model. This is the step people consistently skip, and it is the one that decides whether the agent survives contact with real users. I wrote about why agents fail in production after passing testing, and clean test conversations are the whole story. Testers ask questions they already know the agent can answer. Real users do not.

Step 5: Publish to Teams and Watch the First Week

Publish, then add the Microsoft Teams channel. Start with “available to specific users” and pick a pilot group of ten people, not the whole org. Then read the transcripts in the Analytics section every day for the first week. Not the satisfaction score. The actual conversations. You will find phrasings you never tested, a refusal that fired too aggressively, and at least one question your knowledge source cannot answer that it obviously should. That first week of transcripts is worth more than any amount of pre-launch testing. If generative answers start overriding your scoped topics during this phase, the fix is covered in Stop Copilot Studio Generative Answers From Hijacking Your Topics.

The Working Agent and the Pitfall That Breaks Most First Builds

Final state: one agent, one scoped job, one SharePoint knowledge source, one flow that creates tickets, published to a Teams pilot group. Microsoft’s five-part recipe maps cleanly onto this. The guide is right that anyone can follow the framework.

The pitfall: skipping step 4 because the happy path worked twice in the test canvas. In my experience the agents that embarrass their builders are never the ones that failed in testing. They are the ones that were never tested against inputs designed to break them. Budget as much time for hostile testing as for building. The building is the easy half.

Frequently Asked Questions

How do I build an AI agent in Copilot Studio for the first time?

Start by writing a tight Instructions prompt that gives the agent one clear job and explicit boundaries, then attach a knowledge source like SharePoint, and connect a single Power Automate flow as a tool. Test it before publishing to a channel like Teams. Keeping the scope narrow on your first build makes it much easier to identify where things go wrong.

What should I write in the Instructions field when creating a Copilot Studio agent?

Instead of writing a vague description, define exactly what the agent can and cannot do, which sources it should use, and how it should respond when it does not know the answer. Keeping the instruction set under 500 tokens helps maintain reliability, as longer prompts can reduce consistency.

When should I turn on general knowledge in Copilot Studio?

Only enable general knowledge if your agent genuinely needs to draw on information beyond your attached sources. For internal use cases like a policy or FAQ agent, leaving it off produces more trustworthy answers because responses are grounded in your own documents and can be cited.

Why does my Copilot Studio agent give incomplete answers from PDF files?

The knowledge indexer in Copilot Studio handles standard PDF and Word files well but struggles with scanned images embedded inside PDFs. If your documents contain scanned pages rather than real text, the agent may only read the portions it can process and skip the rest without flagging it.

Source: How to build an AI agent: A simple guide for anyone (Microsoft AI Blog).

August 10, 2026
How to Design Tool Definitions for Copilot Studio Agents

By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published August 6, 2026

The most common failure I see in agent builds is not the model and not the orchestrator. It is the tool definition. Copilot Studio agent tools get selected by a runtime relevance score, not by a human reading a list, and a definition written like internal documentation produces silent misrouting. This walkthrough takes one tool from a vague definition to one the planner can bind against reliably. By the end you have a repeatable checklist you can apply to every tool on every agent.

If you want the mechanics behind why this works, I covered the scoring behavior in how Copilot Studio tool selection works under the hood. This post is the practical half.

Step 1: Name the tool as a verb phrase

Rename the tool from a system name to an action. SAPQuery tells the planner nothing. GetOpenInvoicesForCustomer tells it exactly which user intents should route here. The name is scored alongside the description, so it is free relevance signal. Do not waste it on the name of the backend system. The trade-off: verb-phrase names are longer and less pretty in your solution explorer. Accept that. You are optimizing for the planner, not the maker portal.

Step 2: Write the description as a retrieval hint

This is where most Copilot Studio agent tools go wrong. The orchestrator scores your description against the user’s actual words, so write it in the vocabulary users type, not the vocabulary your architecture diagram uses.

Bad: Calls the finance API endpoint to retrieve invoice records from the ERP integration layer.

Good: Returns unpaid or overdue invoices for a specific customer. Use when the user asks about outstanding balances, open invoices, what a customer owes, or payment status.

The second version contains the phrases real users produce. Anthropic’s guidance on tool use for Claude lands on the same principle: describe when to use the tool, not how it is implemented. This holds across every function calling stack I have worked with, not just Copilot Studio. If you are thinking about how topic descriptions interact with the same orchestration layer, Stop Copilot Studio Generative Answers From Hijacking Your Topics covers the overlap and where the fix lives.

Step 3: Give parameters binding-friendly names and descriptions

A parameter named id of type string gives the planner nothing concrete to bind against. When the user says “show me what Contoso owes,” the planner has to guess which fragment of the utterance belongs in id. Rename it to customerAccountNumber and describe it precisely: The customer's account number in the format CUST-00000. If the user gives a company name instead, resolve it with SearchCustomerByName first.

That last sentence matters. Parameter descriptions are one of the few places you can teach the planner tool sequencing without writing orchestration logic. Use enums wherever the value set is closed. status as a free string invites hallucinated values; an enum of open, paid, overdue removes the guesswork entirely. The Microsoft Learn documentation for Copilot Studio covers the input schema options, but it will not tell you to write them this defensively. Experience will.

Step 4: Return full state, not a status flag

Design the output for the agent’s next turn, not for a log file. A tool that returns {"status": "success"} leaves the agent with nothing to reason about. If the user then asks “which of those is overdue by more than 30 days,” the agent has no data and either calls the tool again or makes something up. Return the invoice list with dates, amounts, and statuses in the response. Yes, that costs tokens. It is cheaper than a second tool call and far cheaper than a confident wrong answer. I traced several production incidents back to exactly this pattern when I wrote about why agents fail in production after passing testing. The same principle applies at the agent architecture level — as I argued in The Model Is the Cheapest Part of Every Enterprise AI Agent, grounding data and tool design are where reliability is actually won or lost.

Step 5: Test with overlapping intents, not happy paths

Do not test by asking the question the tool was built for. Test the boundaries between tools. If you have GetOpenInvoicesForCustomer and GetOrderHistoryForCustomer, ask something ambiguous like “what’s the situation with Contoso’s account” ten times and watch which tool fires. If routing flips between runs, your descriptions overlap and you need to sharpen the “use when” clauses until they carve up the intent space cleanly. Also test with the wrong entity format on purpose. Give a company name where an account number is expected and confirm the planner chains through the lookup tool instead of stuffing garbage into the parameter.

The finished state

You end up with a tool that has a verb-phrase name, a description written in user vocabulary with explicit “use when” phrases, typed parameters with formats and enums, and an output rich enough to support follow-up questions. Apply the same pass to every tool on the agent and misrouting drops sharply. In my experience, this single pass fixes more agent reliability problems than any prompt engineering on the instructions.

One pitfall to watch: do not keep adding well-designed tools forever. Past roughly 10 to 15 tools on a single agent, the relevance signal gets noisy no matter how clean your definitions are. At that point the fix is splitting the agent, not another description rewrite.

Frequently Asked Questions

How do I write good descriptions for Copilot Studio agent tools?

Write descriptions using the vocabulary your users actually type, not technical or architectural language. Include phrases like ‘use when the user asks about…’ so the orchestrator can match the tool to real user intents. Describing when to use a tool, rather than how it works internally, consistently improves routing accuracy.

Why does my Copilot Studio agent keep routing to the wrong tool?

Tool selection is driven by a relevance score calculated at runtime, not a manual lookup, so vague or technical definitions cause silent misrouting. If your tool names and descriptions are written for developers rather than the planner, the orchestrator cannot reliably match them to user input. Rewriting names as verb phrases and descriptions in plain user language usually resolves the problem.

What is the best way to name tools in a Copilot Studio agent?

Name tools as verb phrases that describe a clear action, such as GetOpenInvoicesForCustomer rather than SAPQuery. The name is scored alongside the description, so it carries relevance signal and should reflect user intent rather than the underlying system or API.

How do I define parameters so a Copilot Studio agent can fill them correctly?

Give each parameter a descriptive name and a clear explanation of what value it expects and where that value comes from in the conversation. A parameter named ‘id’ with no context forces the planner to guess, while a name like ‘customerName’ with a description reduces binding errors and makes extraction from user utterances more reliable.

About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn

August 6, 2026
Stop Copilot Studio Generative Answers From Hijacking Your Topics

By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published August 5, 2026

I keep seeing the same complaint in the community: someone builds a solid topic in Copilot Studio, with an adaptive card, a Power Automate action, exact wording that legal signed off on. It works for weeks. Then generative orchestration gets switched on and suddenly Copilot Studio generative answers take over. The agent paraphrases a SharePoint document instead of running the topic. Same question, wrong path.

The usual reaction is to pile on more trigger phrases. That does nothing. Under generative orchestration, trigger phrases are not how routing works anymore.

The problem with generative answers stealing topic-based answers

With classic orchestration, topics fire on trigger phrase matching. With generative orchestration, the model picks between your topics, your tools, and your knowledge sources by scoring their descriptions against the user’s message. Trigger phrases become a weak secondary signal at best.

Most topics I look at have no description, or an auto-generated one like “This topic handles expense questions.” That is a losing hand. Your knowledge source is a 40-page policy PDF full of relevant vocabulary. The orchestrator scores “how do I get reimbursed for a client dinner” against that PDF and against your one-line description, and the PDF wins. Generative answers fire, your scripted topic sits idle, and the user gets a paraphrase instead of the exact approved flow.

It is the same silent misrouting failure mode I wrote about with multi-agent orchestration routing. Nothing errors. Nothing logs a warning. The agent just confidently takes the wrong path.

The fix

Write the topic description as a retrieval hint, not documentation. Open the topic, click Details in the top bar, and fill the Description field. The orchestrator reads this field to decide routing, so it needs to contain the words users actually type.

Weak version:

Handles expense-related questions.

Version that wins routing:

Use this topic for any question about submitting expense reports, reimbursement, expense limits, receipts, client dinners, travel costs, or mileage claims. Always use this topic instead of searching documents when the user asks how to claim or submit an expense.

Two things matter here. First, the vocabulary overlap: name the concrete nouns users will type, because the scoring is lexical-ish relevance, not comprehension. Second, the explicit priority instruction at the end. “Always use this topic instead of searching documents” is blunt, and blunt works. Microsoft documents this behavior in the generative orchestration docs, but the priority-instruction trick is the part most people miss.

If you want to understand exactly how the orchestrator scores and selects between topics and tools, How Copilot Studio Agent Tool Selection Actually Works Under the Hood walks through the scoring pass in detail and explains why descriptions carry so much weight.

One trade-off to be honest about. A description that broad will pull in some borderline questions that generative answers would have handled fine. For compliance-sensitive flows, that is the trade you want. Exact approved wording beats a fluent paraphrase every time. Before you ship any of this, it is also worth running a Power Platform licensing audit to confirm the connectors and flows your topic triggers are covered under the right licenses for every user who will hit this path.

Test it with five phrasings you did not use in the description. If the topic fires on all five, you are done. If not, add the missing nouns and retest. In my experience this takes ten minutes and fixes ninety percent of “my topic stopped working” reports.

About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn

August 5, 2026
The Model Is the Cheapest Part of Every Enterprise AI Agent

By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published July 31, 2026

Every agent conversation I get pulled into starts with the same debate. Opus 5 or GPT. Copilot Studio’s default model or bring-your-own. Someone has a benchmark chart open before anyone has asked what the agent is supposed to do. After more than a year of building enterprise AI agents internally, I am convinced this is the least consequential decision in the entire project. Swapping the model has never taken me more than an afternoon. The data plumbing, the connector policy sprawl, the tool selection behavior, and figuring out whether the thing produces useful work per dollar have each eaten weeks. This post is my map of where the real cost centers sit, and it links out to everything I have written on each layer, so treat it as the index.

Every Enterprise AI Agent Kickoff Starts With the Wrong Debate

Model differences are real. I wrote about why Opus 5’s long-horizon task completion numbers matter more than coding benchmarks, and Anthropic shipping verticalized agents like Claude Finance tells you the frontier labs know where the value sits. But here is the asymmetry nobody prices in: the model is the one component you can replace without touching anything else. Copilot Studio lets you change it in a dropdown. A custom orchestration layer changes it in one config line. Meanwhile your Dataverse schema grounding, your DLP policies, your tool descriptions, and your measurement pipeline are all bespoke work that survives every model swap and cannot be bought off a pricing page.

So when a kickoff burns two hours on model selection, I now say the same thing every time. Budget this project like 10 percent model and 90 percent platform. Every agent I have seen die in month two died in the 90 percent.

Grounding Data Is Where the Real Engineering Hours Go

The hardest part of an internal agent is not intelligence. It is getting the agent to behave like someone who actually works at your organisation. Org policies, naming conventions, tribal knowledge, the exception the finance team makes every quarter-end. None of that is in any model’s training data.

The naive fix is stuffing everything into the system prompt. I learned the hard way that prompts past a few hundred tokens start degrading reliability, and a focused 400-token instruction set beats a 4000-token one every time. That is why I consider business skills in Dataverse an architectural shift rather than a feature: policy moves out of the prompt into a managed record with an owner and a version history. If you want persistent AI output that stays queryable alongside that structured data, Prompt Columns in Dataverse are worth understanding as a companion pattern — you define the prompt once and the result lives as a column rather than a one-off generation.

Same logic applies to schema. An agent that does not know your column logical names burns tokens guessing, which is why the Dataverse MCP server’s metadata inspection tools matter, and why grounding coding agents in live environment metadata is the only reliable fix for hallucinated column names. If you want to understand what is actually happening on the wire when an agent calls one of these tools, I broke down how MCP works under the hood. It is a stateful JSON-RPC protocol, not a REST connector, and treating it like one breaks in production.

Knowledge sources carry the same trap at a different layer. Wire the same SharePoint site into four agents four different ways and you get silent misrouting nobody can diagnose. I covered the governance side in my post on knowledge sources and agent feedback loops. None of this work goes away when you change models. All of it decides whether the agent is trustworthy.

Connector Policies and Tool Selection Decide What Your Agent Can Actually Do

The second cost center is everything between the agent and the systems it touches. I hit this building a GitHub integration: the connector sat in the Business data group in Dev and the Blocked group in Prod, so an agent that passed UAT would not even load its connection at go-live. The full story is in my post on enterprise environment sprawl, and the lesson generalises. The connector is never the problem. The environment strategy around it is.

Then there is tool selection, which most teams treat as magic. It is not. In Copilot Studio, the orchestrator scores tool names, descriptions, parameter names, and enum values against the current turn. It is a retrieval problem, not a dispatch problem, and I walked through the mechanics in how Copilot Studio agent tool selection actually works. A vague operation summary like GetData on a custom connector is not lazy metadata. It is a bad prompt that makes the agent pick the wrong action.

This is also why I push back on premature multi-agent architectures. A model that hits 95 percent tool selection accuracy with two connectors can drop to 70 percent with five, because descriptions start competing with each other. Multi-agent is a tax you pay when a single agent can no longer do the job, not an upgrade, and orchestration routing just moves the misrouting failure one layer deeper where it is harder to debug.

If You Cannot Measure Useful Work Per Dollar, You Built a Demo

The last cost center is the one almost everyone skips. If you cannot define what counts as useful work for your agent in one sentence, it does not have a clear enough job to be worth scaling. I documented my full approach in measuring useful work per dollar for AI agents: a single boolean per invocation, sourced from downstream behavioral signals like ticket reopens rather than thumbs-up ratings, divided into a cost stack of model tokens plus per-message consumption plus premium connector actions.

Measurement is also your early warning system. Agents that sailed through UAT fail in production because test conversations are too clean, and the input distribution shifts the moment real users show up with messy phrasing. A workflow that completes 70 percent of the time is not a workflow, it is a demo, because the human check-in that makes it reliable eliminates the time savings it was supposed to create. Without a useful-work metric you do not find that out until someone quietly stops using the agent.

The encouraging part: this layer is finally getting platform support. Feedback stored as queryable Dataverse rows, closed-loop learning through the Power Apps MCP server, and certified MCPs giving platform teams a trust-signal list instead of shadow adoption from random repos. The tooling is catching up to the problem.

So here is the payoff. Next time an agent project kicks off, do not open with the model comparison. Open with four questions. Where does the org knowledge live and who owns it. Which connectors and policies apply in the environment where this will actually run. How many tools before selection accuracy degrades. And what single boolean defines useful work. In my experience, the teams that answer those four before touching a model dropdown ship agents that survive month three. The teams that start with the benchmark chart ship demos.

Frequently Asked Questions

What is the most expensive part of building enterprise AI agents?

The biggest costs in enterprise AI agents are not the model itself but the surrounding layers: data grounding, governance policies, tool configuration, and measurement pipelines. These are bespoke to your organisation and cannot be swapped out the way a model can, which is why they consume the most engineering time and budget.

How do I choose the right AI model for an enterprise agent?

Model selection matters far less than most teams assume, since modern platforms let you swap models with minimal effort. A better approach is to spend the majority of your planning budget on data grounding, connector policies, and measurement before you finalise any model choice.

Why does my enterprise AI agent give unreliable answers despite a strong model?

Unreliable outputs are usually a grounding problem, not a model problem. If the agent lacks access to your organisation’s policies, naming conventions, and internal context, even a frontier model will produce generic or incorrect responses. Moving that knowledge out of long system prompts and into structured, versioned records tends to improve consistency.

When should I move AI policy instructions out of the system prompt?

Once your system prompt grows beyond a few hundred tokens, reliability starts to drop and maintenance becomes difficult. Migrating policy instructions into a managed data layer gives each rule a clear owner and version history, and it keeps the prompt focused on behaviour rather than organisational context.

About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn

July 31, 2026
How MCP Actually Works Under the Hood When an Agent Calls a Tool
By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published July 28, 2026

Most people treat MCP as the thing that lets their agent call tools, as if it were a fancier function-calling wrapper. That mental model gets you through demos and breaks the moment you try to run something real. If you want to understand how MCP works under the hood, you have to stop thinking about it as a plugin format and start thinking about it as a stateful client-server protocol built on JSON-RPC 2.0.

Towards Data Science published a solid walkthrough of the protocol recently, MCP Explained: How Modern AI Agents Connect to the Real World, and it lines up with what I have been reading in the official Model Context Protocol spec. This post is my attempt to compress the mechanism into a mental model you can actually use when you design servers for Power Platform agents.

The Surface Behaviour: What You See When an Agent Calls an MCP Tool

From the outside, an MCP call looks trivial. You wire an MCP server into Copilot Studio or a Power Apps agent, the agent picks a tool, arguments get filled in, a result comes back, and the model uses it in the next turn. It feels indistinguishable from a REST connector.

That is the illusion. Underneath, three things are happening that a REST connector never does: the client and server negotiated capabilities before any tool call happened, the server is exposing tools, resources, and prompts as separate primitives, and the connection is a live session, not a stateless request. The agent is not just calling an endpoint. It is holding a conversation with a server that has told it what it can do.

Underneath: JSON-RPC, the Handshake, and Capability Negotiation

MCP rides on JSON-RPC 2.0. Every message is a JSON object with a method, params, and either an id (for requests) or no id (for notifications). Nothing exotic. What matters is the sequence.

When an MCP client connects to a server, the first thing it sends is an initialize request. This carries the client’s protocol version and its capabilities. The server responds with its own protocol version, its capabilities, and metadata like server name and version. The client then sends an initialized notification to confirm the handshake is done. Only after this does tool discovery happen, usually through a tools/list call.

Capability negotiation is the piece people miss. The server advertises whether it supports tools, resources, prompts, sampling, logging, and so on. The client picks up only what it understands. This is why an MCP server built against a newer spec can still work with an older client. Backward compatibility is baked into the protocol.

Then there are the three primitives:
- Tools are model-invoked. The agent decides when to call them.
- Resources are application-controlled. The host app decides what to expose.
- Prompts are user-invoked. Templates the user can trigger.
Most people conflate all three into tools. That is why their MCP servers feel bloated. If a chunk of data is really a reference document, it should be a resource, not a tool. Tools are for actions with side effects or computed answers.

Edges and Limits: Transports, Session State, Schema Drift, and Auth

MCP is transport-agnostic in principle but the two real options are stdio and HTTP with Server-Sent Events (or the newer streamable HTTP transport). Stdio is fine for local dev. For anything hosted, you are on HTTP, and that opens the usual questions about session identity, reconnection, and load balancing.

Session state is the first sharp edge. An MCP session is stateful. If your server is behind a load balancer with no sticky sessions, you will see broken initialize sequences and mysterious method not found errors after a reconnect. This is not a bug. It is the protocol working as designed against infrastructure that assumed statelessness.

Schema drift is the second. Tool schemas are exchanged at tools/list time. If your server changes a tool’s input schema mid-session without notifying the client via notifications/tools/list_changed, the agent will keep calling the old shape and fail silently or noisily depending on how strict your validation is. The new Dataverse MCP server tool shape is a good example of what a clean split between metadata inspection, querying, and search actually looks like in practice.

Auth is the third. The core MCP spec deliberately left auth as a transport concern for a long time. The newer authorization spec pins it to OAuth 2.1. If you are wiring an MCP server into an enterprise agent, this is where you spend most of your time, not on the tool logic itself.

What This Means When You Build an MCP Server for Power Platform Agents

A few concrete implications for anyone pointing Copilot Studio or Power Apps agents at MCP.

Design tool descriptions as retrieval hints, not documentation. I wrote about this in how Copilot Studio agent tool selection actually works. The orchestrator scores names, descriptions, and parameter names together. MCP does not change that. It just gives you a cleaner surface to expose them from.

Split tools, resources, and prompts properly. If you dump everything as tools, you inflate the tool count, degrade selection quality past ten to fifteen entries, and force the model to reason about things that should have been passive context.

Treat session lifetime as part of your design. Idle timeouts, reconnect logic, and schema change notifications are not edge cases. They are the load-bearing parts of a production MCP server. The same session and state management issues are a core reason AI agents fail in production when they worked fine in testing.

MCP is not a plugin format. It is a protocol. Build servers like protocol servers and the sharp edges stop cutting. Skip that and you are building another silo with extra JSON on top. In my day-to-day, that shift in framing is what changes the design.

Frequently Asked Questions

How does MCP work under the hood when an agent calls a tool?

MCP is a stateful client-server protocol built on JSON-RPC 2.0, not a simple function-calling wrapper. Before any tool call happens, the client and server go through a capability negotiation handshake, the server declares what it supports, and only then does tool discovery take place. The connection stays live as a session rather than being a stateless request like a REST call.

What is the MCP handshake and why does it matter?

When an MCP client connects to a server, it sends an initialize request containing its protocol version and capabilities, and the server responds with its own. The client then sends an initialized notification to confirm the exchange before any tools are listed or called. Skipping this step or misunderstanding it is a common reason real implementations break outside of demos.

What is the difference between MCP and a standard REST connector?

A REST connector is stateless, meaning each request stands alone with no shared context. MCP maintains a live session, negotiates capabilities upfront, and exposes tools, resources, and prompts as distinct primitives. This makes it better suited for agents that need to understand what a server can do before deciding how to use it.

Why does MCP use JSON-RPC 2.0 instead of a simpler format?

JSON-RPC 2.0 gives MCP a lightweight but structured message format that distinguishes requests from notifications and pairs responses to the correct request using an id field. This structure supports the sequenced, session-based communication that MCP requires, while remaining simple enough to implement across different clients and servers.

Source: Attention Required! (Towards Data Science).

About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn
July 28, 2026
Anthropic’s Claude Opus 5: Why Those Long Horizon Agent Numbers Matter

By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published July 24, 2026

Anthropic shipped Claude Opus 5 and the headline everyone is going to quote is the coding score. Fine. That is not what I am reading twice. The Claude Opus 5 release numbers I keep going back to are the long-horizon agentic task results. If the model can actually hold state and reason across a multi-step, tool-using workflow for meaningfully longer than Opus 4.8 could, that changes what I would trust an agent to do end to end.

I have been running Opus 4.8 through Copilot Studio and a few MCP-connected setups for months. The tail failures I wrote about when 4.8 landed did not disappear. They just moved further out into the workflow. Opus 5 looks like it is aimed directly at that problem.

What it actually does

Opus 5 is the new frontier model in the Claude lineup. The release ships with three things I care about.

First, a jump on long-horizon agentic benchmarks. Anthropic is publishing task-completion numbers on multi-turn, tool-using evals that run significantly longer than the ones they highlighted for 4.8. The distinction matters because average benchmark performance and the ability to keep a coherent plan across 30 or 40 tool calls are not the same skill.

Second, better tool-use reliability. Structured output adherence, function argument shape, and schema compliance are all up. This is the boring plumbing that decides whether your agent works or spends half its runs retrying because it put a string where an integer belonged.

Third, coding scores went up again. This is the number that will dominate every roundup post this week. It is real and it is useful, but for anyone building agents it is downstream of the first two.

The pricing sits in the same neighborhood as Opus 4.8. Available on the direct Anthropic API, on Bedrock, and on Vertex.

Why the long-horizon numbers matter more than the coding score

Here is the thing I keep repeating to people at other organizations who ask about model swaps. A workflow that completes 70 percent of the time is a demo, not a workflow. The reason most internal agents get pulled after the pilot is that the human check-in halfway through is what actually makes them work, and once you add that check-in you have not saved anyone any time.

Long-horizon task performance is the specific number that decides whether you can remove that check-in. If Opus 5 completes a 20-step research-and-write workflow at 85 percent instead of 60, that is not a marginal improvement. That is the difference between a pattern I would deploy and one I would kill. If you are thinking about how to measure whether that improvement actually shows up in practice, How to Measure Useful Work Per Dollar for Your Power Platform AI Agents is the framework I would reach for first.

The coding score is downstream of this for a simple reason. Better tool-use reliability and better structured output are the same underlying capability that makes coding scores go up. A model that produces cleaner function calls also produces cleaner code. So if the coding number moved but the long-horizon number did not, I would be skeptical. Both moved. That is a good signal.

What I am watching for over the next two weeks is whether the published numbers survive contact with real workflows. Anthropic ran a public rollback on Fable 5 earlier this year and the postmortem was clear about how launch evals can miss regressions that only show up in production. Opus 5 will get the same treatment from anyone paying attention.

What I would do with it this week

Three concrete things.

One, pick the longest agent workflow I already have running on Opus 4.8 and rerun the last 50 production traces on Opus 5. Same prompts, same tools, same inputs. Diff the completion rates and the tool-call error rates. Not average latency. Completion rate on the tail. The failure modes worth hunting are exactly the ones I covered in Why Do AI Agents Fail in Production When They Worked Fine in Testing?

Two, look at the system prompt. Every time a new model lands I see people paste the old prompt in and complain the results are worse. Opus 5 will have different steering. I would strip the prompt down to the shortest version that still enforces the policy and let the model do more of the work.

Three, for the Copilot Studio side, wait. Model availability inside Copilot Studio lags the direct Claude API by weeks. There is no point speculating about which slot Opus 5 will fill until Microsoft actually plugs it in. Test on the direct API now, plan the migration path later. I learned the hard way that prototyping on one endpoint and moving to another at the last minute burns a sprint.

If the long-horizon numbers hold up under real traces, this is the release that finally lets me push a handful of agents from human-in-the-loop to human-on-the-loop.

Source: Introducing Claude Opus 5 \ Anthropic (Anthropic).

About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn

July 24, 2026
How to Measure Useful Work Per Dollar for Your Power Platform AI Agents

By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published July 15, 2026

OpenAI published a piece on managing AI investments in the agentic era and the metric they anchor on is useful work per dollar. Not tokens per second. Not model benchmarks. Useful work per dollar for AI agents, measured against the actual business outcome the agent is supposed to produce. That framing maps cleanly onto Power Platform agents if you wire up the telemetry properly. This walkthrough shows how to instrument a Copilot Studio agent so you can see cost, task success rate, and useful work per dollar per workflow, then use that to decide what to scale and what to kill.

The working result: a Dataverse table plus a Power BI page that gives you a per-agent, per-workflow view of cost efficiency in production. No guessing, no vibes.

Step 1: Define what counts as useful work for the agent

This is the step everyone skips and it is the one that decides whether the whole exercise is worth anything. Useful work is not “the agent responded.” It is the outcome the workflow is supposed to produce.

For a support triage agent, useful work might be “ticket classified correctly and routed without human correction within 24 hours.” For an invoice extraction agent, it is “line items extracted with zero manual edits before posting.” For an internal knowledge agent, it is “user did not open a follow-up ticket on the same topic within 48 hours.”

Write this down as a single boolean per invocation: isUsefulWork = true/false. If you cannot define it in one sentence, the agent probably does not have a clear job yet.

Step 2: Log invocations and outcomes to Dataverse

Create a Dataverse table called AgentInvocation with these columns at minimum: InvocationId, AgentName, WorkflowName, UserId, StartTime, EndTime, InputTokens, OutputTokens, ToolCallsCount, Outcome (choice: Success/Failure/Escalated), IsUsefulWork (bool), and Notes.

In Copilot Studio, add a Power Automate flow at the end of the agent topic that writes a row to this table on every invocation. Pass the run context so you can correlate it later. For the IsUsefulWork field, you have two options: infer it automatically from a downstream signal (was the ticket reopened, did the invoice post, did the user thumbs-down) or capture it with a lightweight feedback prompt at the end of the conversation. I prefer downstream signals because self-reported feedback is noisy.

Microsoft has a good primer on writing to Dataverse from flows if you have not done this pattern before.

Step 3: Pull token and action costs into the same table

Cost is not just tokens. For a Copilot Studio agent using Power Automate flows, tool calls, and premium connectors, the cost per invocation is a stack: model tokens plus per-message consumption plus any premium connector actions triggered inside tool flows.

Build a simple cost model as a Dataverse calculated column or a Power BI measure:

InvocationCost = (InputTokens * InputRate) + (OutputTokens * OutputRate) + (ToolCallsCount * FlowActionCost) + MessageMeterCost

Get the current rates from your tenant’s Power Platform licensing page and your model provider. The exact rates will drift, so store them in a separate PricingRates table with an effective date rather than hardcoding them into the measure. Trust me on this one. I learned the hard way when connector pricing shifted and every dashboard I had was quietly wrong for a month.

Step 4: Build the useful work per dollar view in Power BI

Connect Power BI to the AgentInvocation table. Create three core measures:

TotalCost = SUM(AgentInvocation[InvocationCost])

UsefulWorkCount = CALCULATE(COUNTROWS(AgentInvocation), AgentInvocation[IsUsefulWork] = TRUE)

UsefulWorkPerDollar = DIVIDE([UsefulWorkCount], [TotalCost])

Build one page with a matrix visual: rows are WorkflowName, columns are the three measures plus task success rate and average invocation cost. Add a line chart showing useful work per dollar trending weekly per workflow. That trend line is the one you will actually look at.

Slice by agent, by workflow, by user segment. The interesting patterns are usually in the segments, not the overall number. One workflow can be printing money while another quietly burns budget on retries.

Step 5: Set a scale or kill threshold and review weekly

Numbers without a decision rule are decoration. Set a threshold before you look at the data so you do not rationalize whatever the chart shows.

My rule of thumb: if a workflow’s useful work per dollar is at least 3x the manual baseline (what the same work would cost in human time), it is a scale candidate. If it is below 1x for four weeks running, it gets killed or redesigned. Between 1x and 3x is the improvement zone: look at where the cost concentrates, usually retries, oversized system prompts, or unnecessary tool calls. If you want to understand what is driving those unnecessary tool calls, How Copilot Studio Agent Tool Selection Actually Works Under the Hood covers the orchestrator scoring pass and how to structure tools so the agent stops reaching for the wrong one.

Review weekly. Monthly is too slow, daily is noise.

Final state and one pitfall

You end up with a Dataverse table capturing every invocation, a cost model that reflects real pricing, and a Power BI page that shows useful work per dollar per workflow with a scale-or-kill threshold. That is the whole loop.

The common pitfall: measuring useful work by proxies that are easy to log rather than the ones that actually matter. “User did not thumbs-down” is not the same as “the work was correct.” Spend the extra effort to wire up a downstream signal, even if it means waiting 24 hours to mark an invocation useful. Noisy signals produce confident wrong decisions, and this is a place where being confidently wrong is expensive. If you want to see what happens when agents that looked fine in testing start failing on real signals, Why Do AI Agents Fail in Production When They Worked Fine in Testing walks through exactly that. For more on how I think about this stuff, see my LinkedIn.

Frequently Asked Questions

How do I measure useful work per dollar for AI agents in Power Platform?

Start by defining a clear, binary outcome for each agent workflow, such as whether a ticket was routed correctly or an invoice posted without edits. Then log each invocation to Dataverse with cost and outcome data, and use Power BI to calculate the ratio of successful outcomes to total spend. This gives you a per-agent, per-workflow view of cost efficiency rather than relying on vague performance indicators.

What is useful work in the context of a Copilot Studio agent?

Useful work refers to the specific business outcome an agent is designed to produce, not simply the fact that it generated a response. For example, a support triage agent produces useful work only if the ticket is classified and routed correctly without human correction. If you cannot define that outcome in a single sentence, the agent likely lacks a clear purpose.

How do I track AI agent invocation costs in Dataverse?

Create a Dataverse table that records key details for every agent run, including token counts, tool calls, timestamps, and the workflow outcome. A Power Automate flow triggered at the end of each Copilot Studio topic can write this data automatically. Over time, this log gives you the raw numbers needed to calculate cost per successful outcome.

When should I use downstream signals instead of user feedback to measure agent success?

Downstream signals, such as whether a ticket was reopened or an invoice posted without edits, are generally more reliable than asking users to rate the interaction directly. Self-reported feedback tends to be inconsistent and easy to ignore, while automated signals reflect actual business outcomes. Use feedback prompts only when no suitable downstream signal exists for the workflow.

Source: How to manage AI investments in the agentic era (OpenAI).

About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn

July 15, 2026
SharePoint Copilot Apps Just Hit Public Preview and the Structured UX Layer Is What Caught My Eye

By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published July 9, 2026

Microsoft shipped SharePoint Copilot Apps public preview this week on the Microsoft 365 Developer Blog. The pitch is simple: guided, action-oriented business experiences that live inside the Microsoft 365 Copilot flow, combining natural language reasoning with structured UX, validation, permissions, and deterministic operations. Source is here.

I read the post twice. Not because it is dense, but because this fills a gap I have been complaining about internally for months.

What it actually does

A SharePoint Copilot App is a declarative app definition that sits inside SharePoint and surfaces inside Microsoft 365 Copilot as a first-class action target. When a user asks Copilot to do something that matches the app’s intent, Copilot hands off to the app, which then renders a structured form or guided flow with validated inputs, honors SharePoint permissions, and executes deterministic operations against SharePoint data.

So instead of Copilot free-forming a response and hoping the model picked the right list columns, the app enforces the shape of the operation. Required fields are required. Dropdowns are dropdowns. Permissions are the permissions the underlying SharePoint site already has. The reasoning layer stays in Copilot. The execution layer is deterministic.

It is not a Power App. It is not a Copilot Studio agent. It is a SharePoint-native app model that Copilot knows how to invoke.

Why it matters

Pure natural-language Copilot interactions are great for exploration and terrible for repeatable business actions. I have written about this before in the Copilot in Power Apps post: coherent is not the same as correct. When an agent writes a record based on a chatty back-and-forth, you get answers that sound complete and quietly did the wrong thing.

The pattern I keep seeing is teams reaching for a full Copilot Studio agent when what they actually need is a form with three fields, a validation rule, and a write to a SharePoint list. That is overbuild. It is also brittle, because now you have a system prompt, tool definitions, and a Power Automate flow all trying to model behavior that a simple structured input would have handled correctly the first time. If you want to understand why that brittleness compounds once the agent hits production, Why Do AI Agents Fail in Production When They Worked Fine in Testing covers exactly that failure pattern.

SharePoint Copilot Apps land right in that gap. If the process is SharePoint-heavy, the data already lives there, and the action is well-shaped, this is closer to your use case than a custom agent. You get the Copilot entry point without paying the tax of building deterministic execution on top of a probabilistic layer.

The trade-off is real. You are locked into SharePoint as the substrate. You give up the flexibility of a Copilot Studio agent that can orchestrate across multiple systems. And you inherit whatever SharePoint permission model you already have, which is a gift or a curse depending on how tidy your sites are.

There is also the honest question of where this sits against Power Apps. A canvas app already does structured input against SharePoint. What Copilot Apps add is the Copilot invocation surface. Users do not have to know the app exists or navigate to it. They say what they want in Copilot and the app appears in the flow. That is the actual delta.

What I would do with it this week

I would pick one high-volume, low-complexity SharePoint process. Something like a request submission, a status update, or a document check-in with metadata. The kind of thing that today is either a clunky SharePoint form or a Power App that nobody uses because they forget the URL.

Then I would rebuild it as a SharePoint Copilot App and see two things. First, does Copilot route to it reliably when users describe the intent in their own words. Second, does the structured form catch the mistakes that a free-form Copilot conversation would have quietly committed. Both of those are testable in a day.

If routing is unreliable, that is the same failure mode I flagged in the multi-agent orchestration post. Descriptions matter more than trigger phrases. Write them like documentation, not marketing copy. The same principle applies inside Copilot Studio — How Copilot Studio Agent Tool Selection Actually Works Under the Hood is worth reading alongside this if you want to understand how the orchestrator scores and selects actions.

For deeper Copilot extensibility context, the Microsoft 365 Copilot extensibility docs are the reference I would keep open while building. And if you want to compare notes on where this fits in the stack, I am usually posting about it on LinkedIn.

I want to see how the routing behaves under real user language before I commit to a pattern, but this is the first structured action surface inside Copilot that feels appropriately scoped rather than oversized.

Source: SharePoint Copilot Apps Now in Public Preview: From Intent to Action in Microsoft 365 Copilot (Microsoft 365 Developer Blog).

About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn

July 9, 2026
Why Do AI Agents Fail in Production When They Worked Fine in Testing?

By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published July 8, 2026

Short answer: AI agents fail in production because test conversations are too clean, tool descriptions are too vague, and nobody logs the decisions the agent actually makes. The agent looked fine in testing because you asked it the questions you already knew it could answer. Real users do not do that.

This is the question I keep getting from people at other organisations who just deployed their first Copilot Studio agent or LangGraph build and watched it fall over in week one. So here is the longer version.

The longer answer on why AI agents fail in production

Three things go wrong, and they usually go wrong together.

1. The input distribution shifts the moment real users show up. In testing you type “what is the status of order 12345”. In production someone types “hey can you check where my thing from last tuesday is at, i think it was for the warehouse team”. The orchestrator now has to route ambiguous phrasing across multiple topics or tools, and it picks one confidently and gets it wrong. I wrote about this specific failure mode in how Copilot Studio agent tool selection actually works under the hood. The planner is scoring, not dispatching. Noisy input equals noisy scores.

2. Tools return status instead of state. A tool that returns {"result": "done"} gives the agent nothing to reason about on the next turn. When something goes sideways the agent cannot recover because it does not know what actually happened. I have hit this the hard way. The fix is boring: tools return the state the agent needs to make the next decision, not a success flag.

3. Nobody logged the decisions. Run logs tell you the flow executed. They do not tell you why the agent picked tool A over tool B, or why it summarised the ticket that way. Without a decision log you are debugging blind. You end up guessing at prompts.

There is also a fourth thing that gets ignored. Silent failure. A Power Automate action inside a Copilot Studio topic can fail and the agent will still generate a confident-sounding response for work that was never done. The user gets an answer. Nothing throws. You find out three weeks later when someone asks why the record was never created.

How to fix it

Start with the tool contracts. Every tool your agent can call needs a precise name, a description written as a retrieval hint rather than documentation, and parameters with concrete names. customerAccountNumber not id. If you are on Copilot Studio, the Microsoft Learn docs cover the schema but not the hint-writing style. That part you learn by breaking things.

Then cap the tool count. Past roughly 10 to 15 tools on a single agent, selection quality degrades because the relevance signal gets noisy. Split the agent before you add another tool.

Log the decision, not just the execution. For every turn: what did the user say, what tools did the orchestrator consider, what did it pick, what did the tool return, what did the agent do with that. Store it. Query it. This is the only way to improve the policy over time. I go deeper on this in the post on decision ownership.

Add adversarial test cases. Not “does it work”. Test misspellings, mixed languages, requests that sit between two topics, requests that reference something from two turns ago, requests where the user gives incomplete information and expects the agent to ask. This is where production breaks. Test it before production does. If you want a structured way to think about adversarial evaluation, the red team methodology in Anthropic’s jailbreak safeguards framework for Fable transfers directly to internal agent testing.

Fail loudly. If a tool call fails, the agent should know and say so, not paper over it. Wire actual error handling into every Power Automate step the agent can invoke. Return the error state to the orchestrator so it can decide what to do next.

Related gotchas

Two more worth flagging.

Context window drift. Long conversations quietly push earlier turns out of context. The agent forgets what the user told it in turn one by turn twelve. If your use case has long sessions, either summarise state into a persistent variable or split the interaction. I have written more about when a conversational interface is even the right choice in Copilot Studio is not always the answer.

Model updates. The underlying model gets updated. Your prompts that worked last month behave differently this month. This is not theoretical. Anthropic and OpenAI both ship model updates that shift behaviour on edge cases. The Anthropic Fable 5 redeploy postmortem is a useful template for understanding how regressions surface and what a proper rollback and recovery process looks like. If you are running on Claude or any hosted model, version-pin where you can and regression-test when you cannot.

None of this is glamorous. It is the boring work that separates an agent that survives in production from one that gets rolled back in week two. I talk about more of these lessons on LinkedIn if you want to compare notes.

Frequently Asked Questions

Why do AI agents fail in production after passing all their tests?

AI agents often fail in production because test inputs are too clean and predictable compared to what real users actually type. Other common causes include tools that return vague responses the agent cannot reason about, and a lack of decision logging that makes it nearly impossible to diagnose what went wrong.

How do I stop my AI agent from choosing the wrong tool for a user request?

Write tool descriptions as retrieval hints rather than documentation, and use specific parameter names that clearly signal their purpose. Ambiguous phrasing from real users causes the agent to score tools incorrectly, so tighter tool contracts reduce misrouting significantly.

What is silent failure in an AI agent and why does it matter?

Silent failure happens when an action inside an agent workflow fails but the agent still generates a confident-sounding response, giving the user no indication anything went wrong. This is dangerous because the underlying task was never completed, and the problem may not surface until days or weeks later.

When should I add decision logging to an AI agent?

Decision logging should be built in before the agent goes live, not added after something breaks. Without it, you can see that the agent ran but not why it chose a particular tool or produced a specific response, which makes debugging little more than guesswork.

About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn

July 8, 2026
How Copilot Studio Agent Tool Selection Actually Works Under the Hood

By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published July 7, 2026

Most people building agents think copilot studio agent tool selection works like this: you attach a few tools, write a description for each, and the LLM reads the list and picks the right one. That is directionally correct and completely misses what is actually happening at runtime. The orchestrator runs a planning pass. It scores your tools against the current turn. Descriptions, input schemas, and even the order of your tools all feed that scoring.

Once you see the mechanism, you stop writing tool descriptions like documentation and start writing them like retrieval hints. That changes how you name inputs, how many tools you attach to one agent, and when you split an agent instead of adding a fourteenth tool.

What you see from the maker portal

In the maker studio, you attach a tool to an agent. You give it a name, a description, and an input schema (either from a connector, a Power Automate flow, an MCP server, or a prompt). At runtime, you type a message, the agent thinks for a moment, and calls one of the tools. The trace view shows you which tool was picked and what inputs were passed.

That surface makes it look like the model reads the list top to bottom and picks the best match. It is not that simple. The trace hides the planning pass, and the planning pass is where 80% of the reliability of your agent lives. Microsoft’s generative orchestration docs hint at this but do not spell it out in the way a builder needs.

What the orchestrator is actually doing

Between the user turn and the tool call, the orchestrator does something closer to retrieval than dispatch. It takes the current turn, the conversation history, and the agent instructions, and it scores each attached tool for relevance. The scoring uses the tool name, description, input parameter names, parameter descriptions, and enum values if present. Tools with vague names and thin descriptions score badly regardless of how logically correct they are.

Then the planner picks a candidate tool, resolves inputs from the turn context (or asks the user for missing ones), and invokes. If the invocation fails or the result is empty, the planner may retry with a different tool. That retry loop is where token budget disappears and latency creeps up.

This is the same pattern you see in the Copilot Studio release plans that describe how tools and knowledge sources get grounded per turn. It is a retrieval problem wearing a routing costume.

Where the mechanism breaks down

Three failure modes show up over and over. I wrote about the schema version of this in the Dataverse MCP server tool shape post, but it applies to every tool surface.

Overlapping descriptions. Two tools both say something like “Get information about an order.” The planner cannot tell them apart at the description layer, so it falls back to parameter matching, which is noisier. You get silent misrouting where the agent confidently picks the wrong tool.

Vague input schemas. A parameter called id of type string tells the planner nothing. A parameter called customerAccountNumber with a description like “6-digit customer account, not the order number” gives the planner something to bind against.

Long tool lists. Once you attach more than roughly 10 to 15 tools, scoring quality degrades. The signal gets noisy. This mirrors what happens when you stuff too much into a system prompt, which I covered in the business skills post.

How to build once you know this

Write tool descriptions as retrieval hints, not documentation. State what the tool does, when to use it, and critically, when not to use it. “Use this to look up an order by its order number. Do not use this for customer profile lookups.” That negative clause is doing work.

Name parameters like a human would search for them. orderNumber beats id. Add a description on every parameter, even the obvious ones. Enum values are gold because they narrow the planner’s search space to something concrete.

Cap tool count per agent. If you find yourself attaching a fourteenth tool, split the agent by domain and use multi-agent orchestration to route between them. A focused agent with 6 well-described tools outperforms a monster agent with 20 tools every time. The same principle applies when deciding where your agent reads its data from — something I break down in SharePoint vs Dataverse as a Copilot Studio Knowledge Source.

Test the routing, not just the tools. Write a set of representative user turns and check which tool the planner picks. If two tools tie or the wrong one wins, fix the descriptions before you touch the model or the instructions. That is the fastest debugging loop I have found, and it is one I keep coming back to in my own work.

The mechanism is not magic. It is retrieval with extra steps. Once you build for that, your agents get more predictable, cheaper to run, and easier to explain to a stakeholder.

Frequently Asked Questions

How does Copilot Studio agent tool selection actually work at runtime?

Rather than simply reading a list of tools and picking the best match, the orchestrator runs a planning pass that scores each tool based on relevance to the current turn. It factors in the tool name, description, input parameter names, and any enum values before selecting a candidate and resolving the required inputs. This scoring process is closer to retrieval than traditional routing.

Why does my Copilot Studio agent keep picking the wrong tool?

Vague tool names and thin descriptions are the most common cause, as the orchestrator scores tools for relevance and poorly described tools rank badly even if they are logically the right choice. Writing descriptions as retrieval hints rather than documentation, and being specific with parameter names, will improve selection accuracy significantly.

How do I write better tool descriptions for a Copilot Studio agent?

Instead of writing descriptions like reference documentation, treat them as signals that help the orchestrator match the tool to user intent. Be specific about what the tool does, what inputs it expects, and when it should be used rather than a similar tool. Precise parameter names and enum values also feed into the scoring process.

When should I split one Copilot Studio agent into multiple agents instead of adding more tools?

As the number of attached tools grows, the scoring pass has more candidates to evaluate and the risk of the wrong tool being selected increases. If you find yourself adding a large number of tools to a single agent, splitting responsibilities across multiple agents can improve reliability and reduce latency caused by retry loops.

About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn

July 7, 2026

Tag: Copilot Studio

Step 1: Define One Job Before You Build an AI Agent in Copilot Studio

Step 2: Attach a Knowledge Source the Agent Can Cite

Step 3: Give It One Tool With a Power Automate Flow

Step 4: Test in the Canvas With the Activity Map Open

Step 5: Publish to Teams and Watch the First Week

The Working Agent and the Pitfall That Breaks Most First Builds

Frequently Asked Questions

How do I build an AI agent in Copilot Studio for the first time?

What should I write in the Instructions field when creating a Copilot Studio agent?

When should I turn on general knowledge in Copilot Studio?

Why does my Copilot Studio agent give incomplete answers from PDF files?

Step 1: Name the tool as a verb phrase

Step 2: Write the description as a retrieval hint

Step 3: Give parameters binding-friendly names and descriptions

Step 4: Return full state, not a status flag

Step 5: Test with overlapping intents, not happy paths

The finished state

Frequently Asked Questions

How do I write good descriptions for Copilot Studio agent tools?

Why does my Copilot Studio agent keep routing to the wrong tool?

What is the best way to name tools in a Copilot Studio agent?

How do I define parameters so a Copilot Studio agent can fill them correctly?

The problem with generative answers stealing topic-based answers

The fix

Every Enterprise AI Agent Kickoff Starts With the Wrong Debate

Grounding Data Is Where the Real Engineering Hours Go

Connector Policies and Tool Selection Decide What Your Agent Can Actually Do

If You Cannot Measure Useful Work Per Dollar, You Built a Demo

Frequently Asked Questions

What is the most expensive part of building enterprise AI agents?

How do I choose the right AI model for an enterprise agent?

Why does my enterprise AI agent give unreliable answers despite a strong model?

When should I move AI policy instructions out of the system prompt?

The Surface Behaviour: What You See When an Agent Calls an MCP Tool

Underneath: JSON-RPC, the Handshake, and Capability Negotiation

Edges and Limits: Transports, Session State, Schema Drift, and Auth

What This Means When You Build an MCP Server for Power Platform Agents

Frequently Asked Questions

How does MCP work under the hood when an agent calls a tool?

What is the MCP handshake and why does it matter?

What is the difference between MCP and a standard REST connector?

Why does MCP use JSON-RPC 2.0 instead of a simpler format?

What it actually does

Why the long-horizon numbers matter more than the coding score

What I would do with it this week

Step 1: Define what counts as useful work for the agent

Step 2: Log invocations and outcomes to Dataverse

Step 3: Pull token and action costs into the same table

Step 4: Build the useful work per dollar view in Power BI

Step 5: Set a scale or kill threshold and review weekly

Final state and one pitfall

Frequently Asked Questions

How do I measure useful work per dollar for AI agents in Power Platform?

What is useful work in the context of a Copilot Studio agent?

How do I track AI agent invocation costs in Dataverse?

When should I use downstream signals instead of user feedback to measure agent success?

What it actually does

Why it matters

What I would do with it this week

The longer answer on why AI agents fail in production

How to fix it

Related gotchas

Frequently Asked Questions

Why do AI agents fail in production after passing all their tests?

How do I stop my AI agent from choosing the wrong tool for a user request?

What is silent failure in an AI agent and why does it matter?

When should I add decision logging to an AI agent?

What you see from the maker portal

What the orchestrator is actually doing

Where the mechanism breaks down

How to build once you know this

Frequently Asked Questions

How does Copilot Studio agent tool selection actually work at runtime?

Why does my Copilot Studio agent keep picking the wrong tool?

How do I write better tool descriptions for a Copilot Studio agent?

When should I split one Copilot Studio agent into multiple agents instead of adding more tools?