Halilcan Soran

Tag: AI Agents

  • Most Agentic Workflows Are Just Fancy If/Then Logic in a Trench Coat

    Most Agentic Workflows Are Just Fancy If/Then Logic in a Trench Coat

    People keep asking in the community what makes an agentic workflow actually useful. The honest answer is that most things being called agentic workflows right now are not. They are linear automations with a language model bolted on for the response step. That distinction matters more than most teams realise when they start building.

    What a Useful Agentic Workflow Actually Does

    A useful agentic workflow does something a standard Power Automate flow cannot: it makes decisions mid-execution based on context it discovered during the run, not based on conditions you hard-coded before it started.

    That sounds obvious. It is not, in practice.

    A flow that checks a field value and routes left or right is not an agent. An agent is something that can retrieve information it did not start with, reason about what that information means for the current task, and take a different action than you would have anticipated when you designed it. The key word is discovered. The agent had to go and find out something, then act on it.

    If you can fully diagram the execution path before the workflow runs, it is probably not agentic. It is a well-structured flow. There is nothing wrong with a well-structured flow. But you should not be paying the overhead of agent infrastructure to build one.

    Where Teams Go Wrong Building Agentic Workflows

    The most common mistake I see is treating the language model as the agent. The LLM is not the agent. The LLM is the reasoning layer. The agent is the system that decides when to call what tool, handles what comes back, and determines whether the result is good enough to proceed or whether it needs to try something else.

    When that orchestration layer is weak or missing, you get a workflow that calls one tool, takes the output at face value, and moves on. That is not reasoning under uncertainty. That is a glorified lookup with a friendly response message.

    I wrote about silent action failures in the context of Copilot Studio earlier (the production testing post covers this in detail). The same failure mode appears in agentic workflows, but it is worse because the agent has more steps where it can silently accept a bad result and keep going. A flow fails at a specific action. An agent can propagate a bad intermediate result through three more steps before anything looks wrong.

    The Two Things That Make or Break an Agentic Workflow

    Based on what I have built internally and what I hear from people at other organisations, it comes down to two things.

    First: tool design. The actions available to your agent need to return enough context for the agent to evaluate them, not just a success or failure signal. If your Power Automate flow returns {"status": "done"}, the agent has no way to assess whether done means what the user needed. It will treat it as success. Your tools need to return structured, interpretable output. This is not a language model problem. It is an API design problem.

    Second: failure handling that is explicit, not optimistic. A useful agent knows when it is stuck and does something about it. That might mean escalating to a human, asking the user for clarification, or stopping cleanly with an honest message. What it does not do is generate a confident-sounding response for a task that did not complete. That is the failure mode that destroys trust in agents faster than anything else, because the user finds out later, not immediately.

    I covered how this plays out in Copilot Studio specifically in the post on when Copilot Studio is the wrong choice. But the principle applies regardless of the tooling. An agent that cannot fail gracefully is not useful in production. It is a liability.

    What Agentic Workflows Are Actually Good For

    The use cases where agentic workflows justify their complexity share a few characteristics. The task has multiple possible paths and you cannot enumerate them all upfront. The inputs are unstructured or variable enough that rule-based routing breaks down. The system needs to recover from partial failures without a human in the loop for every edge case.

    Document processing that involves extracting, validating, cross-referencing, and then acting on extracted data is a reasonable fit. Multi-step research tasks where what you search for next depends on what you found are a reasonable fit. Anything where the decision logic changes frequently and hard-coding it into a flow becomes a maintenance problem is worth evaluating. Before committing to that architecture, though, it is worth asking whether the underlying process is actually sound — automating a bad process just makes it fail faster, and agentic workflows are no exception.

    A status check is not a fit. A single-action task triggered by a button is not a fit. Anything you can build cleanly as a Power Automate flow with proper error handling is probably not worth the overhead of an agentic architecture. The orchestration cost is real and the debugging surface is larger.

    The Test I Use

    Before committing to an agentic workflow architecture, I ask one question: does this task require the system to discover something during execution that changes what it does next, and would that discovery be different for different runs?

    If yes, agents are worth the investment. If no, you are adding complexity to solve a problem that a well-built flow could handle, and you will spend more time debugging agent behaviour than you saved on logic design.

    The technology is not the constraint. Knowing what you are actually building is.

    Frequently Asked Questions

    What is an agentic workflow?

    An agentic workflow is an automated process that can discover new information during execution and make decisions based on that context, rather than following a path you fully defined in advance. The key difference from standard automation is that the system reasons about what it finds and adapts its actions accordingly. If you can map out every possible execution path before the workflow runs, it is likely not truly agentic.

    When should I use an agentic workflow instead of a standard automation flow?

    Use an agentic workflow when the task requires mid-run decision-making based on information the system has to go and retrieve, not conditions you can pre-define. If your automation can be fully diagrammed before it starts, a well-structured flow like Power Automate will do the job without the added infrastructure cost of an agent.

    Why does my agentic workflow keep producing bad results without throwing any errors?

    This usually happens when the orchestration layer accepts tool outputs at face value without checking whether the result actually meets the goal. Agents can carry a flawed intermediate result through several steps before anything appears wrong, which makes the failure much harder to trace than a standard flow that breaks at a single action.

    How do I design tools that work properly inside an agentic workflow?

    Your tools need to return enough contextual detail for the agent to evaluate the outcome, not just a status signal like done or success. Without meaningful output, the agent cannot reason about whether the action achieved what the task actually required.

  • Adding Copilot to Your Power App Is Not the Same as Making It Smarter

    Adding Copilot to Your Power App Is Not the Same as Making It Smarter

    Microsoft published a post this week about making business apps smarter by embedding Copilot, app skills, and agents directly into Power Apps. The features are real and some of them are genuinely useful. But I keep seeing teams read announcements like that and immediately open their existing apps to start wiring things in. That is where it goes wrong. Adding Copilot to Power Apps does not make the app smarter. It makes the AI visible. Those are different things.

    What App Skills and Agent Integration Actually Do Under the Hood

    When you expose a Power App as an app skill or embed a Copilot Studio agent into a canvas app, you are giving the AI a surface to operate on. The agent can read context from the app, trigger actions, and return responses into the UI. In theory, the AI bridges what the user needs and what the app can do.

    In practice, the agent is only as capable as what you hand it. It reads data from your app’s data sources. It calls the actions you have defined. It interprets user intent against the topics and instructions you have written. If your data model is inconsistent, your actions are incomplete, or your process logic has gaps, the agent does not compensate for any of that. It just operates on top of it and returns confident-sounding responses anyway.

    I wrote about this problem in a different context when covering why Copilot Studio agents fail in production. Silent action failures are one of the nastiest issues: the agent completes its response, the user thinks something happened, nothing actually did. That risk does not disappear when you move the agent inside a Power App. If anything, it gets harder to spot because users expect the app to be reliable.

    Why the Data Model and UX Structure Matter More Than the AI Feature

    Most Power Apps I have seen built inside large organisations were designed around a specific, narrow workflow. The data model reflects decisions made at the time of build, often under time pressure, often by someone who is no longer on the team. Fields are repurposed. Status columns hold values that mean three different things depending on which team is using them. Lookup tables have orphaned records nobody cleaned up.

    When you put an agent on top of that, the agent queries this data and tries to give useful answers. The answers will be coherent. They will not be correct. Not reliably.

    The UX structure compounds this. Canvas apps built for point-and-click navigation do not automatically become good AI surfaces. If a user can ask the agent to update a record, but the app’s own form has fifteen required fields and three conditional rules that only run client-side, you now have a conflict between what the agent can do via a Power Automate action and what the app enforces through its UI. One of them will win. It will not always be the right one.

    This is the same argument I made about automating a bad process. The automation does not fix the process, it executes it faster and more consistently, including the broken parts. Embedding AI into a poorly structured app works the same way.

    What I Check Before Wiring Any Agent Into an Existing App

    Before I connect anything to a Copilot Studio agent or enable app skills on an existing Power App, I go through a short audit. Not a formal document. Just four questions that save a lot of cleanup later.

    • Is the data model clean enough to query? If the same concept is stored in three different columns across two tables with inconsistent naming, the agent will surface that inconsistency directly to the user. Fix the model first.
    • Are the actions the agent can trigger complete and safe? Every Power Automate flow an agent can call needs proper error handling and a defined failure response. Silent failures inside agent topics are a known problem. If the flow does not return a clear success or failure, the agent cannot respond accurately.
    • Does the app enforce rules that the agent needs to know about? If business logic lives only in Power Fx expressions inside the app’s forms, the agent does not see it. Validation that matters needs to exist at the data layer or inside the flows the agent calls.
    • Is the process the app supports well-defined enough to describe to an AI? If I cannot write a clear system prompt describing what the agent should and should not do in this app, the process is not ready. Ambiguity in the process becomes ambiguity in agent behaviour.

    When Embedding AI in a Power App Is Worth It and When It Is Not

    There are genuinely good cases for this. An app where users regularly need to find records across complex filters is a reasonable candidate. Surfacing a conversational shortcut to navigate a large dataset, trigger a common action, or get a summary of a record without clicking through multiple screens can reduce real friction. I have seen it work well when the underlying data is clean and the scope of what the agent can do is narrow and explicit.

    The cases where it is not worth it yet are more common. An app with inconsistent data. A process with unresolved exceptions. A UX that was never designed with AI interaction in mind. In those situations, embedding an agent creates a new layer of support burden without a proportional benefit.

    I also want to be direct about something I mentioned in my post on when Copilot Studio is the wrong choice: not every interaction benefits from being conversational. Some things in a Power App are faster as a button. The AI control is not always an upgrade on a well-placed filter or a clear form layout.

    The Microsoft announcement covers what these features can do. That is useful to know. But the question worth spending time on is not whether you can add Copilot to your Power App. It is whether the app you have is ready to have AI sitting on top of it. Most of the time, that answer requires more honesty than the feature release notes will prompt you to apply.

    Frequently Asked Questions

    How do I add Copilot to a Power App?

    You can embed a Copilot Studio agent into a canvas app or expose your app as an app skill, giving the AI a surface to read context and trigger actions. However, before doing this, your data model and process logic need to be solid, because the agent will only be as reliable as what you give it to work with.

    Why does adding Copilot to Power Apps not make the app smarter?

    Embedding Copilot makes the AI visible inside your app, but it does not fix underlying problems with your data or logic. If your data model is inconsistent or your actions are incomplete, the agent will still return confident-sounding responses that may not be accurate or reliable.

    What is the difference between an app skill and a Copilot Studio agent in Power Apps?

    An app skill exposes your Power App so an AI can interact with it from outside, while embedding a Copilot Studio agent brings the AI directly into the canvas app interface. Both approaches rely on the same principle: the AI can only work with the data sources and actions you have defined for it.

    When should I consider adding AI features to an existing Power App?

    You should only add AI features once your data model is clean, your process logic is complete, and your app’s actions are properly defined and tested. Layering AI onto a poorly structured app creates a conflict between what the agent can do and what the app enforces, which makes failures harder to detect.

    This post was inspired by Making business apps smarter with AI, Copilot, and agents in Power Apps via Microsoft Power Platform Blog.

  • Copilot Studio Is Not Always the Answer

    Copilot Studio Is Not Always the Answer

    I keep seeing this on LinkedIn and in community forums. Someone describes an internal use case, and the first five replies are all “have you tried Copilot Studio?” The tool has gotten good enough that it has become the reflexive answer to any question involving automation, conversation, or AI. That reflex is causing real problems. Knowing when Copilot Studio is the wrong tool is as important as knowing how to build with it well.

    When Copilot Studio Is the Wrong Tool for the Job

    Most misuse I see falls into one of three situations. The use case is purely transactional. The interaction model is not conversational. Or the team wants a workflow, not an agent.

    If someone needs to submit a form, approve a request, or trigger a process on a schedule, that is Power Automate territory. Putting a conversational interface in front of a single-action task does not make it better. It makes it slower, harder to test, and harder to maintain. Users do not want to type a sentence to do something they could do in two clicks.

    The second situation is harder to spot. Some interactions look conversational but are not. A knowledge base search, a document lookup, a status check. These are point-in-time queries with no real back-and-forth. You could build them in Copilot Studio. You could also build them as a Power Apps canvas app with a simple search interface and ship it in a day with less moving parts and a much more predictable failure surface.

    The Agent Complexity Problem

    There is also a complexity ceiling that teams hit faster than expected. Copilot Studio agents work well when the conversation scope is tight. One domain. A few topics. Defined intents. When someone tries to build a single agent that handles HR queries, IT requests, and finance approvals inside the same session, topic routing starts failing at the edges. I wrote about this in Your Copilot Studio Agent Passed Every Test and Still Failed in Production. When a user’s phrasing sits between two topics, the agent picks one confidently and gets it wrong. The more topics you add, the more edge cases you create, and the harder they are to test systematically.

    The instinct to build one agent that does everything is understandable. It feels cleaner. In practice it produces an agent that does everything poorly and fails in ways that are genuinely difficult to diagnose.

    Where the Wrong Choice Usually Starts

    It usually starts with the framing of the requirement. Someone says “we want a chatbot” and that phrase triggers Copilot Studio before anyone has defined what the interaction actually needs to do. I have seen teams spend weeks building agent topics, writing generative AI prompts, and wiring up Power Automate actions, when what the users actually wanted was a better SharePoint search and a weekly digest email.

    The honest question to ask before opening Copilot Studio is this: does this use case genuinely require back-and-forth conversation, or does it just need to surface information or move data? If the answer is the second one, there is almost always a simpler path.

    This is not a knock on Copilot Studio. The tool is genuinely capable when it fits the problem. Handling multi-turn conversations, routing across complex intent patterns, integrating generative answers with structured actions, those are things it does well. But that capability comes with a real operational cost. There is a topic structure to maintain, system prompts that drift when production data introduces edge cases, Power Automate actions that can fail silently inside a topic and return a confident-sounding response for work that was never done.

    What to Reach for Instead

    Power Apps for anything with a fixed interaction model. Canvas apps are underrated for internal tooling. They give you a defined UI, predictable state, and a clear place to debug when something breaks.

    Power Automate for anything triggered, scheduled, or event-driven. If there is no user in the loop having a conversation, there is no reason for Copilot Studio to be involved. Keep in mind that even straightforward flows can run into issues at scale, as Power Automate throttling limits will break your flow in production under real load if you have not accounted for them.

    SharePoint or Dataverse with a search interface for knowledge retrieval. If users are looking something up, build a search experience, not a conversational one.

    In enterprise environments, the governance overhead of Copilot Studio also matters. You are managing an agent that generates natural language responses. That response quality needs to be reviewed, monitored, and occasionally corrected. Most teams I talk to underestimate this cost until they are three months into production and someone in legal asks why the agent said something it should not have.

    The Right Question Before You Build

    Before any Copilot Studio project starts, the question worth asking is not “how do we build this agent” but “does this use case actually need an agent.” If the answer requires you to stretch the definition of conversation to make it fit, that is a sign to stop and pick the simpler tool.

    Copilot Studio is a good tool. It is not a default. Using it where it fits produces something worth building. Using it where it does not produces something you will be maintaining and explaining for a long time.

    Frequently Asked Questions

    When should I use Copilot Studio instead of another tool?

    Copilot Studio works best when the interaction is genuinely conversational, scoped to a single domain, and involves a defined set of intents. If the task is transactional, point-in-time, or better served by a simple form or search interface, tools like Power Automate or Power Apps are likely a faster and more maintainable choice.

    What is the difference between Copilot Studio and Power Automate?

    Power Automate is built for workflow and process automation, such as form submissions, approvals, and scheduled triggers. Copilot Studio is designed for conversational agent experiences. Using Copilot Studio for single-action tasks adds unnecessary complexity without improving the user experience.

    Why does my Copilot Studio agent keep routing users to the wrong topic?

    Topic routing breaks down when an agent is built to handle too many domains or intents within a single session. When a user’s phrasing falls between two topics, the agent will confidently pick one and get it wrong. Keeping each agent focused on a narrow scope reduces these edge cases and makes failures easier to diagnose.

    How do I know if my use case actually needs a chatbot?

    Start by defining what the interaction needs to do before choosing a tool. If users need a back-and-forth conversation to complete a task, a conversational agent may be appropriate. If they need a search result, a status update, or a simple action, a canvas app or improved search interface will often deliver a better outcome in less time.

  • Your Copilot Studio Agent Passed Every Test and Still Failed in Production

    Your Copilot Studio Agent Passed Every Test and Still Failed in Production

    I came across a post from Zapier Blog about AI agent evaluation, and it described something I keep seeing inside large organisations: an agent that looks perfect in a demo, gets signed off, goes live, and then immediately starts doing things nobody expected. Wrong tool calls. Conversation loops that never resolve. Outputs that look confident and are completely wrong. The post frames this well as a sandbox problem. But the fix it describes, better test coverage and smarter metrics, only gets you partway there. The deeper issue with Copilot Studio agent testing is not the quantity of your tests. It is what you are actually testing for.

    Why Demo-Passing Agents Break in Real Workflows

    When a team builds an agent in Copilot Studio, they test it against the happy path. A user asks a clean question. The agent triggers the right topic or action. The response looks good. Someone in the review meeting says it works great. The agent gets promoted to production.

    The problem is that real users do not ask clean questions. They ask incomplete ones. They switch intent halfway through a conversation. They paste in text that includes formatting your prompt never anticipated. They use your agent for things it was never designed to do, because nothing in the interface tells them not to.

    None of that shows up in a demo. It shows up three days after go-live when someone forwards you a conversation log that reads like a stress test you forgot to run.

    The Three Failure Modes I Keep Seeing in Copilot Studio Agents

    After building and reviewing a number of agents internally, the failures cluster into three patterns.

    Topic misrouting at the edges. Your agent routes correctly when the user says exactly what you expected. But natural language is messy. When a user’s phrasing sits between two topics, the agent picks one confidently and gets it wrong. You only discover this when someone captures a failed session and traces it back. By then, a dozen other users have hit the same wall and just stopped using the agent.

    Action failures that degrade silently. A Power Automate flow or a connector action fails in the background and the agent carries on as if nothing happened. No error surfaced. No fallback triggered. The user gets a response that implies the task completed. It did not. This is the agent equivalent of a flow that retries quietly and masks the problem until the load goes up. I wrote about that pattern in the context of Power Automate throttling limits breaking flows under real load. The same logic applies here: silent success is not success.

    Prompt instruction drift under real data. Your system prompt was written against clean test data. Production data is not clean. It has unexpected characters, long strings, mixed languages, or values that push the model toward an interpretation you did not intend. The agent’s behaviour drifts. Not catastrophically. Just enough to become unreliable in ways that are hard to reproduce and harder to explain to stakeholders.

    How to Build a Behavioral Test Suite Instead of an Output Checklist

    Most teams build an output checklist. Did the agent return the right answer for these ten questions? That tells you almost nothing about production behaviour.

    What you actually need is a behavioral test suite. The difference is this: output testing checks what the agent said. Behavioral testing checks how the agent handled the situation.

    Here is how I approach it inside Copilot Studio before promoting anything to production.

    Build adversarial input sets, not just representative ones. For every topic your agent handles, write three versions of the trigger: the clean version, an ambiguous version that could belong to two topics, and a broken version with incomplete or oddly formatted input. If the agent routes all three correctly, you have something worth shipping. If it fails on the ambiguous case, you have a routing gap that will hit real users constantly.

    Test conversation state, not just single turns. Copilot Studio agents hold context across a conversation. Test what happens when a user changes their mind on turn three. Test what happens when they ask a follow-up that assumes context the agent should have retained but might not. Single-turn testing misses an entire class of failure that only appears in multi-turn sessions. This is also why agentic workflows require a fundamentally different design approach, not just an AI layer placed on top of existing processes.

    Inject real data samples into action inputs. Pull a sample of actual data from your environment and run it through the actions your agent calls. Do not use synthetic test data if you can avoid it. Real data has edge cases your synthetic data will never cover. If your agent calls a flow that queries a SharePoint list, run the query against the actual list with actual entries, including the ones with blank fields and formatting you did not anticipate.

    Define explicit fallback behaviour and test it deliberately. Every agent should have a defined behaviour for when it cannot complete a task. Most teams add a fallback topic and assume it works. Test it by constructing inputs that should trigger it. If the fallback does not fire, or fires on the wrong inputs, fix it before go-live. A graceful failure is far better than a confident wrong answer.

    What to Monitor After Go-Live and When to Pull an Agent Back

    Testing before launch is necessary but not sufficient. Agent behaviour shifts as the inputs it receives in production diverge from what you tested against. You need monitoring in place from day one.

    Track escalation rate and abandon rate per topic. If a topic is seeing significantly higher escalations than others, that is a signal of routing or response quality problems, not user error. Track action failure rates separately from conversation outcomes. An agent can complete a conversation and still have failed to do the thing the user needed.

    Set a threshold before launch. If escalation rate exceeds a number you agree on in advance, or if a specific action is failing more than a defined percentage of the time, you pull the agent back or disable the affected topic. The threshold is arbitrary. Having no threshold at all is not.

    The agents I have seen hold up in production are not the ones with the most sophisticated prompts. They are the ones where someone spent real time on the failure cases before launch and built actual monitoring into the plan from the start.

    If you are still signing off agents based on demo performance, you are not testing. You are hoping.

    Frequently Asked Questions

    Why does my Copilot Studio agent testing pass in demos but fail in production?

    Most Copilot Studio agent testing is built around ideal user inputs and predictable conversation paths, which do not reflect how real users actually behave. In production, users ask incomplete questions, switch intent mid-conversation, and use the agent in unintended ways that no demo ever surfaces. Testing needs to go beyond the happy path to catch these edge cases before go-live.

    What are the most common failure modes in Copilot Studio agents?

    The three patterns that appear most often are topic misrouting when user phrasing falls between two intents, action failures that complete silently without triggering any error or fallback, and prompt instructions that break down when they encounter messy real-world data. Each of these can go undetected in testing because they only emerge under realistic conditions.

    How do I know if a Power Automate action failed inside my Copilot Studio agent?

    Silent action failures are a serious risk because the agent can continue the conversation and imply a task completed when it did not. You need explicit error handling and fallback logic in your flows so that failures surface to the user rather than being masked by a confident-sounding response.

    When should I test my Copilot Studio agent against real production data?

    You should test against realistic data before promotion to production, not after. System prompts written against clean test data can behave unpredictably when they encounter unexpected characters, mixed languages, or long strings that only appear in live environments. Incorporating a sample of real or representative data into your test suite is a necessary step before sign-off.

    This post was inspired by AI agent evaluation: How to test and improve your AI agents via Zapier Blog.