Halilcan Soran

Tag: AI Agents

  • How to Measure Useful Work Per Dollar for Your Power Platform AI Agents

    How to Measure Useful Work Per Dollar for Your Power Platform AI Agents

    Power BI dashboard showing useful work per dollar for AI agents in Copilot Studio

    By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published July 15, 2026

    OpenAI published a piece on managing AI investments in the agentic era and the metric they anchor on is useful work per dollar. Not tokens per second. Not model benchmarks. Useful work per dollar for AI agents, measured against the actual business outcome the agent is supposed to produce. That framing maps cleanly onto Power Platform agents if you wire up the telemetry properly. This walkthrough shows how to instrument a Copilot Studio agent so you can see cost, task success rate, and useful work per dollar per workflow, then use that to decide what to scale and what to kill.

    The working result: a Dataverse table plus a Power BI page that gives you a per-agent, per-workflow view of cost efficiency in production. No guessing, no vibes.

    Step 1: Define what counts as useful work for the agent

    This is the step everyone skips and it is the one that decides whether the whole exercise is worth anything. Useful work is not “the agent responded.” It is the outcome the workflow is supposed to produce.

    For a support triage agent, useful work might be “ticket classified correctly and routed without human correction within 24 hours.” For an invoice extraction agent, it is “line items extracted with zero manual edits before posting.” For an internal knowledge agent, it is “user did not open a follow-up ticket on the same topic within 48 hours.”

    Write this down as a single boolean per invocation: isUsefulWork = true/false. If you cannot define it in one sentence, the agent probably does not have a clear job yet.

    Step 2: Log invocations and outcomes to Dataverse

    Create a Dataverse table called AgentInvocation with these columns at minimum: InvocationId, AgentName, WorkflowName, UserId, StartTime, EndTime, InputTokens, OutputTokens, ToolCallsCount, Outcome (choice: Success/Failure/Escalated), IsUsefulWork (bool), and Notes.

    In Copilot Studio, add a Power Automate flow at the end of the agent topic that writes a row to this table on every invocation. Pass the run context so you can correlate it later. For the IsUsefulWork field, you have two options: infer it automatically from a downstream signal (was the ticket reopened, did the invoice post, did the user thumbs-down) or capture it with a lightweight feedback prompt at the end of the conversation. I prefer downstream signals because self-reported feedback is noisy.

    Microsoft has a good primer on writing to Dataverse from flows if you have not done this pattern before.

    Step 3: Pull token and action costs into the same table

    Cost is not just tokens. For a Copilot Studio agent using Power Automate flows, tool calls, and premium connectors, the cost per invocation is a stack: model tokens plus per-message consumption plus any premium connector actions triggered inside tool flows.

    Build a simple cost model as a Dataverse calculated column or a Power BI measure:

    InvocationCost = (InputTokens * InputRate) + (OutputTokens * OutputRate) + (ToolCallsCount * FlowActionCost) + MessageMeterCost

    Get the current rates from your tenant’s Power Platform licensing page and your model provider. The exact rates will drift, so store them in a separate PricingRates table with an effective date rather than hardcoding them into the measure. Trust me on this one. I learned the hard way when connector pricing shifted and every dashboard I had was quietly wrong for a month.

    Step 4: Build the useful work per dollar view in Power BI

    Connect Power BI to the AgentInvocation table. Create three core measures:

    TotalCost = SUM(AgentInvocation[InvocationCost])

    UsefulWorkCount = CALCULATE(COUNTROWS(AgentInvocation), AgentInvocation[IsUsefulWork] = TRUE)

    UsefulWorkPerDollar = DIVIDE([UsefulWorkCount], [TotalCost])

    Build one page with a matrix visual: rows are WorkflowName, columns are the three measures plus task success rate and average invocation cost. Add a line chart showing useful work per dollar trending weekly per workflow. That trend line is the one you will actually look at.

    Slice by agent, by workflow, by user segment. The interesting patterns are usually in the segments, not the overall number. One workflow can be printing money while another quietly burns budget on retries.

    Step 5: Set a scale or kill threshold and review weekly

    Numbers without a decision rule are decoration. Set a threshold before you look at the data so you do not rationalize whatever the chart shows.

    My rule of thumb: if a workflow’s useful work per dollar is at least 3x the manual baseline (what the same work would cost in human time), it is a scale candidate. If it is below 1x for four weeks running, it gets killed or redesigned. Between 1x and 3x is the improvement zone: look at where the cost concentrates, usually retries, oversized system prompts, or unnecessary tool calls. If you want to understand what is driving those unnecessary tool calls, How Copilot Studio Agent Tool Selection Actually Works Under the Hood covers the orchestrator scoring pass and how to structure tools so the agent stops reaching for the wrong one.

    Review weekly. Monthly is too slow, daily is noise.

    Final state and one pitfall

    You end up with a Dataverse table capturing every invocation, a cost model that reflects real pricing, and a Power BI page that shows useful work per dollar per workflow with a scale-or-kill threshold. That is the whole loop.

    The common pitfall: measuring useful work by proxies that are easy to log rather than the ones that actually matter. “User did not thumbs-down” is not the same as “the work was correct.” Spend the extra effort to wire up a downstream signal, even if it means waiting 24 hours to mark an invocation useful. Noisy signals produce confident wrong decisions, and this is a place where being confidently wrong is expensive. If you want to see what happens when agents that looked fine in testing start failing on real signals, Why Do AI Agents Fail in Production When They Worked Fine in Testing walks through exactly that. For more on how I think about this stuff, see my LinkedIn.

    Frequently Asked Questions

    How do I measure useful work per dollar for AI agents in Power Platform?

    Start by defining a clear, binary outcome for each agent workflow, such as whether a ticket was routed correctly or an invoice posted without edits. Then log each invocation to Dataverse with cost and outcome data, and use Power BI to calculate the ratio of successful outcomes to total spend. This gives you a per-agent, per-workflow view of cost efficiency rather than relying on vague performance indicators.

    What is useful work in the context of a Copilot Studio agent?

    Useful work refers to the specific business outcome an agent is designed to produce, not simply the fact that it generated a response. For example, a support triage agent produces useful work only if the ticket is classified and routed correctly without human correction. If you cannot define that outcome in a single sentence, the agent likely lacks a clear purpose.

    How do I track AI agent invocation costs in Dataverse?

    Create a Dataverse table that records key details for every agent run, including token counts, tool calls, timestamps, and the workflow outcome. A Power Automate flow triggered at the end of each Copilot Studio topic can write this data automatically. Over time, this log gives you the raw numbers needed to calculate cost per successful outcome.

    When should I use downstream signals instead of user feedback to measure agent success?

    Downstream signals, such as whether a ticket was reopened or an invoice posted without edits, are generally more reliable than asking users to rate the interaction directly. Self-reported feedback tends to be inconsistent and easy to ignore, while automated signals reflect actual business outcomes. Use feedback prompts only when no suitable downstream signal exists for the workflow.

    Source: How to manage AI investments in the agentic era (OpenAI).

    About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn

  • Dataverse July 2026 Update Extends the Agent Data Platform to More Coding Marketplaces and Certified MCPs

    Dataverse July 2026 Update Extends the Agent Data Platform to More Coding Marketplaces and Certified MCPs

    Dataverse July 2026 update expanding agent data platform with certified MCPs

    By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published July 12, 2026

    Microsoft shipped the July 2026 Dataverse update on July 6, and reading it back to back with the previous months makes the pattern obvious. The dataverse july 2026 update is not about new agent features. It is about reach and governance. The Dataverse plugin is expanding into more coding agent marketplaces, MCP tool coverage is growing, partner MCPs are getting certified, and internal MCPs are being pulled under enterprise governance.

    If you have been paying attention to the last three months of Dataverse announcements, this one fits the arc. Microsoft is turning Dataverse into the substrate every agent on the stack talks to, and now they are dealing with the mess that comes with that.

    What shipped in the July 2026 Dataverse update

    Four things landed together.

    First, the Dataverse plugin for coding agents is being pushed into more coding agent marketplaces. This is the same plugin that fixed the hallucination problem for coding agents generating Power Platform code. Wider distribution means more developers will hit it through their existing IDE and agent setups instead of having to hunt for it.

    Second, MCP tool coverage is growing. The Dataverse MCP server now exposes more of the platform surface, which continues the direction we saw when the tool shape shifted toward metadata inspection. Tighter tool boundaries, more of them, less guessing by the agent.

    Third, partner MCPs can now be certified. This is the one I think most teams will underestimate. Microsoft is publishing a certification path so third-party MCP servers can carry a trust signal. Think of it like connector certification, but for the agent tool layer.

    Fourth, internal MCPs get pulled under enterprise governance. DLP, environment scoping, audit. The plumbing you would expect from a Power Platform connector, applied to the MCP tool surface an agent can call.

    Why the certified MCP and governance angle matters

    Here is the part I keep thinking about. Every team I talk to that has been experimenting with agents in the last six months has quietly wired up MCP servers from wherever. Community repos. Vendor previews. Somebody’s GitHub. The tool surface an internal agent can reach is already sprawling, and nobody centralised the decision.

    Sound familiar? It is the same shape as the SharePoint site as a knowledge source problem I wrote about when knowledge sources and feedback loops shipped. Four makers, four wirings, no shared governance. MCP is worse because the surface is executable, not just readable.

    Certified partner MCPs are Microsoft’s answer to shadow adoption. Give platform teams a list of MCPs that carry a trust signal, register the ones that make sense, block the rest. The enterprise governance layer for internal MCPs is the other half. DLP policies that treat MCP tools like any other data movement. If you want to understand how the orchestrator evaluates tools before it ever calls them, how Copilot Studio agent tool selection actually works under the hood is worth reading alongside this update.

    The trade-off is real. Certification takes time. If your team needs a niche MCP that nobody has certified yet, you are either waiting or accepting risk. But the alternative is what I keep seeing on LinkedIn from people six months into agent projects. A tool surface nobody can inventory, and no clean way to answer the auditor when they ask what data the agent can touch. That sprawl is also one of the cleaner explanations for why AI agents fail in production when they worked fine in testing.

    What I would do with it this week

    I would not build a new agent this week. I would audit.

    Start with a simple inventory. Which MCP servers are already wired to any agent in your tenant. Where they came from. Who registered them. Whether they touch Dataverse, SharePoint, or anything with customer data. Most teams have never written this down.

    Then look at the certified partner MCP list as it lands and pick the two or three that actually match what your makers have been asking for. Register those under a governed environment. Publish the list internally. Say the quiet part out loud. If it is not on the list, it does not go into a production agent.

    Finally, get the Dataverse plugin for coding agents into the hands of the pro devs writing plugins and PCF controls. Not the citizen dev crowd. This is a pro dev tool and treating it otherwise misses the point. Scope it to a dev environment with a least-privilege app user, not System Administrator on production. I wrote about that setup in the original plugin post and it still applies.

    If you want to see how this ties back to the broader agent data platform direction, the Power Platform docs are catching up quickly and worth a scan. And if you want to compare notes on how other teams are handling MCP governance, I keep track of what people are saying on LinkedIn.

    The month the platform team gets ahead of MCP sprawl is worth more than the month you shipped one more agent.

    Source: Dataverse Is Your Agent Data Platform: Here’s What’s New in July 2026 (Microsoft Power Platform Blog).

    About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn

  • Anthropic Shipped Reflect With Claude and the Journaling Angle Is More Interesting Than It Looks

    Anthropic Shipped Reflect With Claude and the Journaling Angle Is More Interesting Than It Looks

    Reflect with Claude guided journaling interface on a laptop screen

    By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published July 9, 2026

    Anthropic just shipped Reflect with Claude, a guided journaling and self-reflection experience built on top of the Claude models. On the surface this reads as a wellness feature. Look at it for two more minutes and Reflect with Claude is a positioning move that tells you where Anthropic thinks the interaction pattern is going.

    I want to talk about what it does, why I find it more interesting than the category suggests, and what I plan to actually do with it this week.

    What it actually does

    Reflect is a structured journaling surface inside Claude. You show up, Claude prompts you with reflection questions, and the session builds on prior sessions rather than starting from zero every time. It is opinionated about pacing. It asks follow-ups. It holds context across days.

    The mechanics that matter, if you read past the wellness framing:

    • The model prompts the user, not the other way around. That flips the default chat pattern.
    • Memory persists across sessions in a scoped, purposeful way.
    • The interaction is structured around a specific outcome, reflection, rather than open-ended tasks.
    • Privacy defaults are tighter than a general Claude conversation, which Anthropic calls out directly.

    You can try it on Claude.ai now. It is not an API surface, it is a product experience.

    Why it matters for people who build agents

    The wellness label is the wrapper. The interesting part is that Anthropic is publicly shipping a template for what I would call a companion pattern: model-initiated, memory-aware, purpose-bounded. That is not the same shape as a task assistant.

    Most enterprise agents I see people build are task assistants. User asks, agent does, session ends. That works for a lot of internal automation. It falls apart the moment you want an agent that follows up, checks in, or maintains a thread across days without being re-prompted from scratch every time.

    I have written before about how business logic and memory need real ownership in agent design. Reflect with Claude is a public reference point for what memory-aware, model-initiated interaction feels like when the vendor actually commits to it. The UX decisions in a product like this leak into what users start expecting from every agent they touch, including the ones I ship internally.

    The positioning piece is worth naming too. Anthropic is signaling that Claude is not just a coding model or a knowledge worker copilot. It is a companion product. That is a different market than OpenAI is currently chasing with agents-as-workers, and it is a different market than Microsoft is chasing with Copilot Studio. If you follow the model landscape at all, this is the kind of move that tells you where the roadmap is pointed. I keep tracking these positioning shifts because they change what stakeholders ask for six months later. Anthropic’s approach to red-teaming and safeguards methodology is part of what makes a product like this credible at the enterprise level, and worth studying regardless of whether you are building consumer or internal agents.

    My honest first take: this is more interesting than the launch post makes it sound. The category is wellness, the pattern is architecture.

    What I would do with it this week

    Three things, in order.

    First, I am going to actually use it for a week. Not skim it, use it. I want to feel how the prompting cadence works, how the memory shows up in session two versus session five, and where the friction lives. You cannot design a companion pattern for internal agents if you have never used a shipped one end to end.

    Second, I want to compare the interaction rhythm to what I have tried to build inside Copilot Studio agents. Copilot Studio is very good at request-response. It is not built for a model that opens the conversation. If I want an agent that pings a user with a reflection prompt at the end of the week, what does that actually look like on the Microsoft stack today? Probably a scheduled cloud flow that seeds a Copilot Studio conversation, which is a workaround, not the shape the platform was designed for. Worth stress testing. The reasons agents fail in production often come down to exactly this kind of mismatch between what you prototype and what the platform was actually designed to sustain.

    Third, I want to write down which parts of the Reflect UX I would steal for internal use cases. Not the wellness angle. The structural parts. Session continuity, model-initiated prompts, bounded memory, tight privacy defaults. These are the pieces that generalize to things like weekly project retrospectives, onboarding check-ins, or post-incident reviews with an agent.

    I cannot wait to try this properly. The launch might read as soft, but the shape underneath is the part I want to sit with.

    Source: A new way to reflect on how you use Claude \ Anthropic (Anthropic).

    About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn

  • Why Do AI Agents Fail in Production When They Worked Fine in Testing?

    Why Do AI Agents Fail in Production When They Worked Fine in Testing?

    Why AI agents fail in production despite passing tests

    By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published July 8, 2026

    Short answer: AI agents fail in production because test conversations are too clean, tool descriptions are too vague, and nobody logs the decisions the agent actually makes. The agent looked fine in testing because you asked it the questions you already knew it could answer. Real users do not do that.

    This is the question I keep getting from people at other organisations who just deployed their first Copilot Studio agent or LangGraph build and watched it fall over in week one. So here is the longer version.

    The longer answer on why AI agents fail in production

    Three things go wrong, and they usually go wrong together.

    1. The input distribution shifts the moment real users show up. In testing you type “what is the status of order 12345”. In production someone types “hey can you check where my thing from last tuesday is at, i think it was for the warehouse team”. The orchestrator now has to route ambiguous phrasing across multiple topics or tools, and it picks one confidently and gets it wrong. I wrote about this specific failure mode in how Copilot Studio agent tool selection actually works under the hood. The planner is scoring, not dispatching. Noisy input equals noisy scores.

    2. Tools return status instead of state. A tool that returns {"result": "done"} gives the agent nothing to reason about on the next turn. When something goes sideways the agent cannot recover because it does not know what actually happened. I have hit this the hard way. The fix is boring: tools return the state the agent needs to make the next decision, not a success flag.

    3. Nobody logged the decisions. Run logs tell you the flow executed. They do not tell you why the agent picked tool A over tool B, or why it summarised the ticket that way. Without a decision log you are debugging blind. You end up guessing at prompts.

    There is also a fourth thing that gets ignored. Silent failure. A Power Automate action inside a Copilot Studio topic can fail and the agent will still generate a confident-sounding response for work that was never done. The user gets an answer. Nothing throws. You find out three weeks later when someone asks why the record was never created.

    How to fix it

    Start with the tool contracts. Every tool your agent can call needs a precise name, a description written as a retrieval hint rather than documentation, and parameters with concrete names. customerAccountNumber not id. If you are on Copilot Studio, the Microsoft Learn docs cover the schema but not the hint-writing style. That part you learn by breaking things.

    Then cap the tool count. Past roughly 10 to 15 tools on a single agent, selection quality degrades because the relevance signal gets noisy. Split the agent before you add another tool.

    Log the decision, not just the execution. For every turn: what did the user say, what tools did the orchestrator consider, what did it pick, what did the tool return, what did the agent do with that. Store it. Query it. This is the only way to improve the policy over time. I go deeper on this in the post on decision ownership.

    Add adversarial test cases. Not “does it work”. Test misspellings, mixed languages, requests that sit between two topics, requests that reference something from two turns ago, requests where the user gives incomplete information and expects the agent to ask. This is where production breaks. Test it before production does. If you want a structured way to think about adversarial evaluation, the red team methodology in Anthropic’s jailbreak safeguards framework for Fable transfers directly to internal agent testing.

    Fail loudly. If a tool call fails, the agent should know and say so, not paper over it. Wire actual error handling into every Power Automate step the agent can invoke. Return the error state to the orchestrator so it can decide what to do next.

    Related gotchas

    Two more worth flagging.

    Context window drift. Long conversations quietly push earlier turns out of context. The agent forgets what the user told it in turn one by turn twelve. If your use case has long sessions, either summarise state into a persistent variable or split the interaction. I have written more about when a conversational interface is even the right choice in Copilot Studio is not always the answer.

    Model updates. The underlying model gets updated. Your prompts that worked last month behave differently this month. This is not theoretical. Anthropic and OpenAI both ship model updates that shift behaviour on edge cases. The Anthropic Fable 5 redeploy postmortem is a useful template for understanding how regressions surface and what a proper rollback and recovery process looks like. If you are running on Claude or any hosted model, version-pin where you can and regression-test when you cannot.

    None of this is glamorous. It is the boring work that separates an agent that survives in production from one that gets rolled back in week two. I talk about more of these lessons on LinkedIn if you want to compare notes.

    Frequently Asked Questions

    Why do AI agents fail in production after passing all their tests?

    AI agents often fail in production because test inputs are too clean and predictable compared to what real users actually type. Other common causes include tools that return vague responses the agent cannot reason about, and a lack of decision logging that makes it nearly impossible to diagnose what went wrong.

    How do I stop my AI agent from choosing the wrong tool for a user request?

    Write tool descriptions as retrieval hints rather than documentation, and use specific parameter names that clearly signal their purpose. Ambiguous phrasing from real users causes the agent to score tools incorrectly, so tighter tool contracts reduce misrouting significantly.

    What is silent failure in an AI agent and why does it matter?

    Silent failure happens when an action inside an agent workflow fails but the agent still generates a confident-sounding response, giving the user no indication anything went wrong. This is dangerous because the underlying task was never completed, and the problem may not surface until days or weeks later.

    When should I add decision logging to an AI agent?

    Decision logging should be built in before the agent goes live, not added after something breaks. Without it, you can see that the agent ran but not why it chose a particular tool or produced a specific response, which makes debugging little more than guesswork.

    About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn

  • How Copilot Studio Agent Tool Selection Actually Works Under the Hood

    How Copilot Studio Agent Tool Selection Actually Works Under the Hood

    Diagram of copilot studio agent tool selection at runtime

    By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published July 7, 2026

    Most people building agents think copilot studio agent tool selection works like this: you attach a few tools, write a description for each, and the LLM reads the list and picks the right one. That is directionally correct and completely misses what is actually happening at runtime. The orchestrator runs a planning pass. It scores your tools against the current turn. Descriptions, input schemas, and even the order of your tools all feed that scoring.

    Once you see the mechanism, you stop writing tool descriptions like documentation and start writing them like retrieval hints. That changes how you name inputs, how many tools you attach to one agent, and when you split an agent instead of adding a fourteenth tool.

    What you see from the maker portal

    In the maker studio, you attach a tool to an agent. You give it a name, a description, and an input schema (either from a connector, a Power Automate flow, an MCP server, or a prompt). At runtime, you type a message, the agent thinks for a moment, and calls one of the tools. The trace view shows you which tool was picked and what inputs were passed.

    That surface makes it look like the model reads the list top to bottom and picks the best match. It is not that simple. The trace hides the planning pass, and the planning pass is where 80% of the reliability of your agent lives. Microsoft’s generative orchestration docs hint at this but do not spell it out in the way a builder needs.

    What the orchestrator is actually doing

    Between the user turn and the tool call, the orchestrator does something closer to retrieval than dispatch. It takes the current turn, the conversation history, and the agent instructions, and it scores each attached tool for relevance. The scoring uses the tool name, description, input parameter names, parameter descriptions, and enum values if present. Tools with vague names and thin descriptions score badly regardless of how logically correct they are.

    Then the planner picks a candidate tool, resolves inputs from the turn context (or asks the user for missing ones), and invokes. If the invocation fails or the result is empty, the planner may retry with a different tool. That retry loop is where token budget disappears and latency creeps up.

    This is the same pattern you see in the Copilot Studio release plans that describe how tools and knowledge sources get grounded per turn. It is a retrieval problem wearing a routing costume.

    Where the mechanism breaks down

    Three failure modes show up over and over. I wrote about the schema version of this in the Dataverse MCP server tool shape post, but it applies to every tool surface.

    Overlapping descriptions. Two tools both say something like “Get information about an order.” The planner cannot tell them apart at the description layer, so it falls back to parameter matching, which is noisier. You get silent misrouting where the agent confidently picks the wrong tool.

    Vague input schemas. A parameter called id of type string tells the planner nothing. A parameter called customerAccountNumber with a description like “6-digit customer account, not the order number” gives the planner something to bind against.

    Long tool lists. Once you attach more than roughly 10 to 15 tools, scoring quality degrades. The signal gets noisy. This mirrors what happens when you stuff too much into a system prompt, which I covered in the business skills post.

    How to build once you know this

    Write tool descriptions as retrieval hints, not documentation. State what the tool does, when to use it, and critically, when not to use it. “Use this to look up an order by its order number. Do not use this for customer profile lookups.” That negative clause is doing work.

    Name parameters like a human would search for them. orderNumber beats id. Add a description on every parameter, even the obvious ones. Enum values are gold because they narrow the planner’s search space to something concrete.

    Cap tool count per agent. If you find yourself attaching a fourteenth tool, split the agent by domain and use multi-agent orchestration to route between them. A focused agent with 6 well-described tools outperforms a monster agent with 20 tools every time. The same principle applies when deciding where your agent reads its data from — something I break down in SharePoint vs Dataverse as a Copilot Studio Knowledge Source.

    Test the routing, not just the tools. Write a set of representative user turns and check which tool the planner picks. If two tools tie or the wrong one wins, fix the descriptions before you touch the model or the instructions. That is the fastest debugging loop I have found, and it is one I keep coming back to in my own work.

    The mechanism is not magic. It is retrieval with extra steps. Once you build for that, your agents get more predictable, cheaper to run, and easier to explain to a stakeholder.

    Frequently Asked Questions

    How does Copilot Studio agent tool selection actually work at runtime?

    Rather than simply reading a list of tools and picking the best match, the orchestrator runs a planning pass that scores each tool based on relevance to the current turn. It factors in the tool name, description, input parameter names, and any enum values before selecting a candidate and resolving the required inputs. This scoring process is closer to retrieval than traditional routing.

    Why does my Copilot Studio agent keep picking the wrong tool?

    Vague tool names and thin descriptions are the most common cause, as the orchestrator scores tools for relevance and poorly described tools rank badly even if they are logically the right choice. Writing descriptions as retrieval hints rather than documentation, and being specific with parameter names, will improve selection accuracy significantly.

    How do I write better tool descriptions for a Copilot Studio agent?

    Instead of writing descriptions like reference documentation, treat them as signals that help the orchestrator match the tool to user intent. Be specific about what the tool does, what inputs it expects, and when it should be used rather than a similar tool. Precise parameter names and enum values also feed into the scoring process.

    When should I split one Copilot Studio agent into multiple agents instead of adding more tools?

    As the number of attached tools grows, the scoring pass has more candidates to evaluate and the risk of the wrong tool being selected increases. If you find yourself adding a large number of tools to a single agent, splitting responsibilities across multiple agents can improve reliability and reduce latency caused by retry loops.

    About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn

  • Anthropic’s Fable 5 Rollback: What the Postmortem Reveals

    Anthropic’s Fable 5 Rollback: What the Postmortem Reveals

    Anthropic Fable 5 redeploy postmortem notes on a screen

    By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published July 1, 2026

    Anthropic pulled Fable 5 shortly after its initial release, then redeployed it with fixes and a public writeup. The anthropic fable 5 redeploy is not the story I care about. The postmortem is.

    Rolling back a flagship model publicly is rare. Doing it with a clear explanation of what broke and how they verified the fix is rarer. I have been reading this one twice because it maps almost directly onto how anyone deploying agents internally should think about regressions.

    What Anthropic actually shipped

    Fable 5 is the narrative-tuned variant in the Claude lineup, the one I wrote about when it first landed alongside Mythos 5. The redeploy brings the model back online after Anthropic identified regressions the initial release had introduced, including drift in persona stability and inconsistencies the launch evals had missed.

    The redeployed build ships with expanded regression tests, additional persona stability checks, and a documented verification pass before the rollout was reopened. In the writeup Anthropic names what broke, what they added to catch it next time, and what they changed in their pre-release process. That is the part I keep coming back to.

    You can read the full note on Anthropic’s site. It is short. It is worth your ten minutes.

    Why the rollback and redeploy matters

    Most frontier model releases treat launch as a one-way door. Ship, patch quietly, move on. Public rollback of a flagship variant tells you something about how the company thinks about the contract with the people building on top of it.

    Three things stand out.

    First, the transparency raises the bar. If you are building an agent on top of a model and the vendor tells you exactly what regressed and how they verified the fix, you can decide whether your workflow is affected. If the vendor patches silently, you find out through user complaints. I would rather have the writeup.

    Second, the operational discipline is the template. Naming the regression, adding tests that would have caught it, running a verification pass, and only then reopening the rollout is exactly the pattern I want internal agent deployments to follow. Most internal agent rollouts I hear about from people at other organisations skip at least two of those four steps.

    Third, this raises expectations for the whole space. When one vendor publishes a postmortem like this, the ones that keep patching silently start looking like they have something to hide. That is good pressure on the market.

    The uncomfortable read is what it says about launch evals in general. Fable 5 shipped, passed whatever gates it passed, and still needed a rollback. If that can happen at Anthropic, it is happening everywhere. The difference is whether you hear about it. That same assumption problem shows up in workforce planning too, and the OpenAI EU AI workforce report makes the case that the workflow change bucket is where most of the silent breakage actually lives.

    What I would do with this news this week

    Two concrete things.

    One. Write down what a rollback looks like for the agents you have in production. Not the theory. The actual steps. Who decides to pull it, how you flip the switch, what the fallback behavior is, and how you tell users. If you cannot answer those four in a paragraph, you do not have a rollback plan, you have a wish.

    Two. Look at your regression tests for agent behavior. If the only thing you check before a prompt change or a model swap is whether the happy path still works, you are shipping the same way Fable 5 shipped the first time. Add persona drift checks. Add tool-use reliability checks. Add a few adversarial prompts you know used to fail. This is exactly the kind of discipline I flagged when Opus 4.8 landed, because tail failures are what kill agentic workflows in production.

    If you are running Claude in a Power Automate flow or a Copilot Studio agent, the practical version is simple. Pin the model version in your connector config. Do not auto-upgrade. Keep a small eval set you can rerun on every model change. Before you reach for a desktop flow to automate that verification step, it is worth checking whether a cloud flow would do the job instead. The docs make version pinning straightforward, and it costs you nothing until the day it saves you.

    I have written more on how I think about model selection and agent reliability over on LinkedIn. The short version is this. Every model you depend on will regress at some point. The vendors that tell you about it are the ones worth building on.

    The next flagship release will land soon. The question is whether the postmortem discipline sticks.

    Source: Redeploying Claude Fable 5 \ Anthropic (Anthropic).

    About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn

  • Dataverse MCP Server Got a New Tool Shape and the Metadata Inspection Story Is What Stands Out

    Dataverse MCP Server Got a New Tool Shape and the Metadata Inspection Story Is What Stands Out

    Dataverse MCP server tool shape diagram showing metadata inspection and query tools

    By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published June 25, 2026

    Microsoft published a new post on June 8, 2026 walking through the updated Dataverse MCP server tool shape. The headline is that agents can now inspect metadata, query records, and search across structured and unstructured data through cleaner, well-defined tool boundaries. You can read the original write-up on the Power Platform blog.

    This is not a feature dump. It is a redesign of how the agent talks to Dataverse. And that is more interesting than it sounds.

    What it actually does

    The old tool surface gave agents a handful of broad tools and expected them to figure out the rest from context. In practice, that meant the agent would call a query tool, get back something it could not interpret, call again with different parameters, fail, and burn tokens guessing at column logical names. I have watched this happen in traces. It is not pretty.

    The new shape breaks the surface into clearer categories. Metadata inspection is its own thing now. The agent can ask what tables exist, what columns a table has, what the relationships look like, before it tries to query anything. Record querying sits next to that with predictable inputs. Search across structured and unstructured data is its own tool boundary, so the agent does not have to invent a strategy for whether to hit the relational side or the knowledge side.

    The practical effect: fewer wasted tool calls, less prompt bloat from stuffing schema hints into the system prompt, and more predictable agent behavior when it hits a Dataverse environment cold.

    Why it matters

    This connects directly to a pattern I keep hitting. When you wire an agent to business data, the model is rarely the problem. The friction is the discovery loop. The agent does not know your schema. It does not know your naming conventions. It does not know that your account table has a custom column called cr1a3_segment that nobody documented.

    The old answer was to dump schema into the system prompt. That works until your prompt crosses a few hundred tokens and agent reliability starts degrading. I wrote about this in the context of business skills in Dataverse, and the same logic applies here. Pushing knowledge into a managed, queryable surface beats stuffing it into a prompt every time.

    The new tool shape gives the agent a path to discover the schema on demand instead of carrying it around. That is the architectural shift. It is the same lesson I keep seeing play out across the agent stack: tighter tool boundaries beat broader ones, because they shrink the search space the model has to reason about.

    It also pairs well with the low-latency Dataverse to Fabric sync story. If your agent can inspect metadata cleanly on the operational side and pull near real-time analytical data on the Fabric side, the data path behind the agent finally starts feeling coherent instead of stitched together.

    What I am skeptical about: tool shape changes are easy to undo if the team gets feedback that agents want more flexibility. I hope Microsoft holds the line on the boundaries. Broad, do-everything tools are how you end up back where you started.

    What I would do with it this week

    First, point an agent at a non-production Dataverse environment with the updated MCP server and watch the tool call traces. Not the chat output. The traces. That is where you see whether the new shape actually reduces wasted calls or just renames them.

    Second, strip schema hints out of system prompts on any existing Copilot Studio agent wired to Dataverse, and let the metadata inspection tool do the work. Compare reliability before and after. I expect the shorter prompt wins, based on what I have seen building internally and on what I keep reading from peers at other organisations. The Dataverse plugin for coding agents that Microsoft shipped at Build 2026 is worth reading alongside this, because it tackles the same hallucination-from-missing-schema problem from a different angle. You can also check the Power Platform docs for the current setup steps.

    Third, write down which tool should handle which question type for your agent and treat that as a contract. If the agent starts calling the search tool for things that should hit the query tool, that is a sign your boundary is wrong, not that the tool is wrong. I have been writing about these patterns on LinkedIn for a while now, and the discipline of writing the contract down before deployment saves you a month-six cleanup project.

    If the new tool shape holds up under real workloads, this becomes the default way agents talk to business data on the Microsoft stack.

    Source: Dataverse MCP Server: Understanding the New Tool Shape (Microsoft Power Platform Blog).

    About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn

  • Power Platform June 2026 Update Shipped and the Power Apps MCP Server Closed-Loop Learning Is the Headline

    Power Platform June 2026 Update Shipped and the Power Apps MCP Server Closed-Loop Learning Is the Headline

    Power Platform June 2026 update MCP server closed-loop learning diagram

    By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published June 14, 2026

    The Power Platform June 2026 update MCP server announcement dropped on June 11 and the headline for me is not the usual feature list. It is the Power Apps MCP server and what Microsoft is calling closed-loop learning for enterprise agents. Source: What’s new in Power Platform: June 2026 feature update.

    I read the post twice. The first read I filed it under “another MCP integration.” The second read I realised what they actually shipped. Agents can now learn from real usage signals inside Power Apps without you standing up a data science team to run training and evaluation cycles. That is a different category of thing.

    What it actually does

    The Power Apps MCP server exposes app context, user actions, and outcomes to an agent as structured signals. Until now, teaching an agent how your org works meant one of two paths. Either you fed it knowledge as documents and custom instructions, which is what most teams are doing today. Or you ran your own training, evaluation, and optimization loop, which almost nobody outside the big AI shops actually does.

    Closed-loop learning sits in between. The agent observes what happens when a user takes its suggestion. Did the form get submitted. Did the approval get reversed two days later. Did the user override the recommendation. Those signals flow back through the MCP server and the agent’s behavior adjusts based on whether its suggestions actually worked.

    The mechanism is the interesting part. It is not fine-tuning the underlying model. It is updating the agent’s grounding and skill selection based on outcome data that lives in Dataverse. Combine this with the business skills work Microsoft shipped earlier and the picture gets clearer. Skills are the policy layer. MCP server signals are the feedback layer.

    Why it matters

    Here is the architectural shift I think the release notes underplay. Agent knowledge stops being a build-time artifact and starts becoming a runtime asset.

    Most agents I see today are frozen at deploy time. You write the instructions, attach the knowledge sources, ship it, and then it just sits there making the same mistakes for six months until someone files a complaint and a developer goes in to patch the prompt. That is the model we have been stuck with since Copilot Studio launched.

    Closed-loop learning breaks that pattern. If the signal pipeline works the way the post describes, an agent that recommends the wrong approver three times in a row will stop doing that without anyone editing a prompt. The policy still lives in a business skill with a named owner. The behavior tunes itself based on what actually worked.

    The risk is obvious and I want to name it. If you wire usage signals into agent learning without thinking about which signals count, you end up reinforcing the wrong behavior. A user clicking accept because they are tired is not the same signal as a user clicking accept because the suggestion was right. The MCP server gives you the plumbing. Picking the right outcome signals is still your job, and I expect the first wave of failures to come from teams treating every user action as positive feedback.

    I also want to flag the governance angle. The Power Platform May 2026 update made me start thinking about agent constraints as versioned records. Runtime learning means version history now has to cover learned behavior too, not just authored skills. That is a real audit question and Microsoft has not fully answered it yet.

    What I would do with it this week

    I have an internal approval-routing workflow that I have been wanting to test something like this on. The agent currently picks an approver based on a static skill, and it gets the routing wrong roughly one in six times because the org chart does not reflect who actually owns what.

    My plan for this week is simple. Wire the Power Apps MCP server to that app. Define two outcome signals. First, did the selected approver actually approve within 48 hours, or did they reassign. Second, did the requester come back and edit the routing manually after submission. Anything else I am ignoring on purpose, because I want a clean signal before I open the door wider.

    Then I let it run for two weeks and compare the routing accuracy against the static baseline. If the closed loop moves the needle even five points on a workflow this messy, I will be writing about it. If it does not, I want to understand why before I trust it on anything that matters. If you are thinking about how to expose this kind of agent behavior through a proper UI layer, the guide on building custom Copilot UI widgets in Power Apps is worth reading alongside this.

    The Power Platform docs will need to catch up on the MCP server specifics, and I expect the configuration story to get cleaner over the next few releases. For now, this is the most interesting thing shipped this month and I cannot wait to see what the signal quality looks like in a real workflow. More on this once I have run the experiment. You can follow along on LinkedIn if you want the results.

    Source: What’s new in Power Platform: June 2026 feature update (Microsoft Power Platform Blog).

    About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn

  • Anthropic Dropped Claude Fable 5 Mythos 5 and the Long-Form Narrative Angle Is What Caught Me

    Anthropic Dropped Claude Fable 5 Mythos 5 and the Long-Form Narrative Angle Is What Caught Me

    Claude Fable 5 Mythos 5 release announcement for long-form narrative work

    By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published June 9, 2026

    Anthropic just shipped the Claude Fable 5 Mythos 5 release, a variant tuned specifically for long-form narrative and worldbuilding. You can read the announcement on the Anthropic news page. This is not a flagship capability bump. It is another specialized lineup entry, and the pattern is what caught my attention.

    Finance Agents. Creative. Now Fable and Mythos. Anthropic is not just shipping bigger models. They are shipping shaped ones.

    What it actually does

    Fable 5 Mythos 5 is a Claude variant tuned for long-form fiction, worldbuilding, and narrative consistency. The headline improvements are in three areas.

    First, long-context coherence. Holding a character voice, world rules, and plot threads across tens of thousands of tokens without drift is the actual hard problem in long-form generation. The model is tuned to keep internal consistency, names spelled the same way, magic systems that do not contradict themselves three chapters later, characters that do not forget what they did in chapter two.

    Second, persona stability. Once you give it a narrator voice or a character voice, it holds it. Most general-purpose models start regressing toward a neutral helpful tone after a few thousand tokens. This one is trained to resist that pull.

    Third, structural awareness. It understands scenes, beats, chapter structure, and pacing in a way that is closer to how writers actually think about their work, not as a flat stream of tokens.

    This is not a model for code. It is not a model for tool use. It is a model for people who write novels, run tabletop campaigns, or build interactive narrative experiences.

    Why it matters even if you are not writing fiction

    I am not writing a fantasy novel this week. Most people reading this are not either. So why does this release matter?

    Because the underlying improvements are exactly what agentic systems need.

    Persona stability is the same problem as agent persona stability. If you are building a Copilot Studio agent that needs to talk like an HR specialist or a finance approver, the failure mode is identical: after enough turns, the agent regresses to generic assistant tone. The work Anthropic did to keep a narrator’s voice locked across 50,000 tokens is the same work that keeps your internal procurement agent from drifting into ChatGPT mode by message twelve.

    Long-context coherence is the same problem as multi-turn agent memory. An agent that loses track of what it told the user three turns ago, or contradicts a constraint it acknowledged at the start of the session, is failing the same test a novel writing model fails when it changes a character’s eye color in chapter seven.

    The specialization play is the real signal here. Anthropic is not selling one Claude. They are segmenting by job. As I wrote when Anthropic launched Claude Finance Agents, this is a deliberate move away from raw capability tiers and toward fit-for-purpose variants. That changes how you pick a model. Not “which is smartest” but “which is shaped for my job.”

    This is also a hint about what the next round of enterprise variants will look like. If Fable and Mythos work commercially, expect more shaped Claudes. Legal. Medical. Compliance. The lineup is going to get longer, not shorter. The Anthropic enterprise AI services arm launching to deploy Claude in production is the infrastructure that makes this segmentation commercially viable at scale.

    What I would do with it this week

    Three things, in order.

    One, read Anthropic’s own writeup carefully, not for the fiction angle but for what they say about persona retention and long-context evaluation. The eval methodology is the interesting part. If you build agents, those metrics are closer to what you should be measuring than HumanEval scores.

    Two, run a quick comparison test on Claude. Take an agent system prompt you already have, one that defines a specific persona, and run a long multi-turn conversation on a general Claude variant. Note where the persona breaks. That gives you a baseline. When Fable Mythos style tuning eventually filters into general or enterprise variants, you will know what to measure.

    Three, stop storing persona definitions in 4000-token system prompts. I have written about this before, and it stays true. If Anthropic is doing the persona work at the model level, your job is to give the model a tight, focused instruction set and let the tuning do the rest. A 400-token persona definition with clear constraints beats a 4000-token character bible every time. The same principle applies when you are thinking about whether to build a multi-agent system instead of a single agent: tighter scope and cleaner constraints outperform bloated configurations every time.

    Anthropic is shipping shaped models faster than most teams can decide which one to use. That is a good problem to have.

    Source: Claude Fable 5 and Claude Mythos 5 \ Anthropic (Anthropic).

    About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn

  • Anthropic Shipped Claude Opus 4.8 and the Coding Numbers Are What Caught My Eye

    Anthropic Shipped Claude Opus 4.8 and the Coding Numbers Are What Caught My Eye

    Claude Opus 4.8 release notes shown on a developer screen with agent workflow diagrams

    By Halilcan Soran · Power Platform & AI Agents · Senior Business Analyst at Visa · Published May 28, 2026

    Anthropic dropped the Claude Opus 4.8 release today. I read the post twice before writing anything because the coding and tool-use deltas are the part that actually matters for people building agents, not the headline number.

    Most model bumps are easy to ignore. This one I am paying attention to, and not for the reason most posts will tell you.

    What it actually does

    Opus 4.8 is an incremental release on top of the Opus 4 line. The model card focuses on three areas: coding quality, tool-use reliability, and long-horizon agentic task completion. Anthropic reports gains on SWE-bench style coding tasks and improvements on multi-step tool sequences where the model has to plan, call a tool, read the result, and decide what to do next without losing the thread.

    The pricing and context window stay in line with the Opus 4 family. It is available on the Anthropic API, on Claude.ai, on Amazon Bedrock, and on Google Vertex. Same deployment story as before.

    The thing that jumped out at me was the reduction in tool-call errors on longer agent traces. Not the average. The tail. That is a different problem to fix and a different problem to feel in production.

    Why it matters for agent builders

    If you have built anything resembling a real agentic workflow, you know the failure mode. The agent runs fine for 4 or 5 steps. Then on step 8 it calls a tool with malformed JSON, or hallucinates an argument, or forgets which tool it already called. The whole trace dies and you are looking at logs trying to figure out where the drift started.

    Coding quality and tool-use reliability are the same problem wearing two different hats. A model that writes better code is a model that produces better structured outputs, better function arguments, and better adherence to a schema. That is what makes a multi-step agent stop falling apart on turn 9.

    I have written before about latency being the quiet killer of agentic workflows. Reliability is the louder one. A workflow that completes 70 percent of the time is not a workflow. It is a demo. And most teams I talk to are sitting at 70 percent and calling it production.

    If Opus 4.8 actually pushes the tail of tool-call failures down, that is the difference between an agent you can schedule overnight and an agent that needs a human babysitting every run. That is the gap I care about.

    The other piece worth naming: this is Anthropic doubling down on coding and agents as the wedge. Not consumer chat. Not creative writing. They are picking a lane and shipping into it. Anthropic has been consistent about this for months and Opus 4.8 fits that pattern cleanly. That same strategic focus is visible in moves like Anthropic acquiring Stainless, where the SDK layer itself became a competitive asset.

    What I would do with it this week

    One test. Not a benchmark. A real one.

    Take the agent or automation in your stack that fails most often on long traces. The one where you keep adding retry logic and validation steps because the model keeps producing slightly wrong tool arguments. Run the same trace 20 times on whatever model you are using now. Log the failure points. Swap to Opus 4.8 with the exact same prompt and tools. Run it 20 more times.

    If the tail failures drop noticeably, you have your answer. If they do not, you keep what you have and you saved yourself a migration.

    I would also pay attention to cost per successful completion, not cost per token. A more expensive model that completes the workflow on the first try beats a cheaper model that needs three retries every time. That math is what people forget when they compare price sheets. It is the same logic that applies when you are deciding whether to build a multi-agent system instead of a single agent — the right architecture is the one that actually completes reliably, not the one that looks cleanest on a whiteboard.

    One more thing. If you are running Claude through Bedrock or Vertex, give the model version a week or two to land cleanly in your region before you build anything critical on it. I learned the hard way that prototyping on the direct API and then lifting to Bedrock at the last minute will burn a sprint on region availability and version naming. And if you are watching the broader competitive picture on AI coding agents in enterprise environments, the OpenAI and Dell Codex on-premise partnership is worth reading alongside this release — the two moves are aimed at the same buyer.

    Opus 4.8 is not a revolution. It is a sharpening. And for the kind of long-running agent work I keep seeing teams try to stand up, sharpening is exactly what is needed right now.

    Source: Introducing Claude Opus 4.8 \ Anthropic (Anthropic).

    About the author: Halilcan Soran is a Senior Business Analyst at Visa working on process optimization with Power Platform, RPA, and agentic AI. He previously led a 15-person team at Thomson Reuters, where the Power Automate solutions he built started his automation career. More about Halilcan · LinkedIn