Halilcan Soran

Tag: Enterprise Automation

  • Low-Code Platform Comparisons Miss the Point for Enterprise Power Platform Teams

    Low-Code Platform Comparisons Miss the Point for Enterprise Power Platform Teams

    I came across a post from Zapier Blog ranking the best low-code automation platforms, and it reminded me of a conversation I keep having with stakeholders. Someone reads a roundup, sends it over, and asks why we are not using one of the other tools on the list. The question sounds reasonable. The comparison is not. For teams doing power platform for enterprise automation, these lists are almost always built around the wrong frame entirely.

    Why Platform Comparison Lists Are Built for Buyers Who Do Not Exist in Enterprise

    Roundups like this are useful for one type of reader: someone at a small company, starting from scratch, with no existing infrastructure, who needs to pick a tool this week. That reader exists. Most people building automation inside a large organisation are not that reader.

    Enterprise teams are not choosing between platforms in a vacuum. They are operating inside a tenant. They have an existing Microsoft 365 agreement. They have an IT security function that has already decided what can touch production data. They have a DLP policy, or they are about to have one. The question is never which platform wins a feature comparison. The question is what is already inside the perimeter and how far can it go.

    When the starting point is a Microsoft 365 E3 or E5 agreement, Power Platform is not an option on a menu. It is largely already there. The conversation is about how deeply to use it, not whether to adopt it at all.

    What These Roundups Get Wrong About How Power Platform Actually Works at Scale

    The comparisons that show up in these lists treat features as equivalent when they are not. They will note that Power Automate supports HTTP connectors, and so does Zapier, so check. They will note that both have flow triggers and conditional logic. Check and check.

    What they do not cover is how governance works when you have hundreds of flows built by dozens of makers across multiple environments. Power Platform has environment-level DLP policies that enforce which connectors can interact with which data classifications. You can block a connector tenant-wide from the admin centre. You can require solution-aware flows before anything goes near a production environment. None of that is a feature you evaluate in a roundup. It is architecture you depend on when something goes wrong at 2am and you need to know exactly what touched what.

    Connector-level governance also ties directly into Entra ID. Service principal authentication, conditional access policies, managed identities for flows that call Azure resources. These are not nice-to-haves. They are what your security team will ask about before any automation touches HR data or finance systems. A platform comparison that does not address this is not comparing the same thing your enterprise is actually buying.

    The Governance and Tenant Boundary Argument Nobody in These Lists Makes

    The argument that actually matters for enterprise teams is about the boundary. Everything inside your Microsoft tenant shares an identity layer, a licensing model, an audit log, and a set of compliance controls. Power Platform lives inside that boundary by design. When a Power Automate flow calls Dataverse, or a Copilot Studio agent hands off to an AI Builder model, or a Power App writes back to SharePoint, none of that crosses a boundary. It is all inside the same governance envelope.

    When you bring in a third-party automation tool, you immediately introduce a boundary crossing. Data leaves the tenant. Authentication has to be managed separately. Your audit trail splits. Your DLP logic does not follow. That is not an argument against ever using other tools. But it is the argument that platform comparison lists never make, because they are not written for people managing compliance obligations across a 10,000-person organisation.

    I have written before about how throttling in Power Automate has two distinct layers, platform-level and connector-level, and understanding which one you are hitting matters. The same principle applies here. There are two distinct layers to platform selection: what the tool can do, and what the tool is allowed to do inside your security perimeter. Most comparison articles only address the first layer.

    How to Respond When a Stakeholder Sends You One of These Articles

    This happens. Someone senior reads a roundup, sees that another tool scored well on ease of use or pricing, and asks a reasonable question. Here is how I handle it.

    First, do not get defensive about Power Platform. That reads as tribal and closes the conversation. Instead, reframe the question. The roundup is answering “which tool is easiest to try”. The enterprise question is “which tool can we govern, audit, and scale without introducing a new identity boundary or violating our data residency requirements”.

    Second, be specific about what already exists. If you have 200 flows in production, connectors pre-approved by security, an admin centre your IT team actually monitors, and makers who already know the platform, the switching cost is not zero. It is very large. That context belongs in the conversation.

    Third, acknowledge what the other tools do well. Zapier is genuinely easier to set up for a simple two-step integration. Make has a visual canvas that some people find clearer than Power Automate’s. Agreeing on the narrow case where another tool wins builds credibility for the broader argument about why it does not win at enterprise scale. The same logic applies when teams start layering AI into their automations: as I explored in Agentic Workflows Are Not Just Fancy Automation, adding an AI layer does not transform a poorly governed process into a reliable one, regardless of which platform you are on.

    The roundup is not wrong. It is just answering a different question. Once you say that clearly, the conversation usually moves to something more useful than defending a platform choice that was effectively made the day the Microsoft agreement was signed.

    Frequently Asked Questions

    Why should enterprises use Power Platform for enterprise automation instead of other low-code tools?

    For most large organisations, Power Platform is already included in their Microsoft 365 agreement, so the decision is less about choosing a tool and more about how deeply to use one that is already available. It also integrates directly with existing Microsoft security infrastructure, including Entra ID, conditional access policies, and tenant-level governance controls that other platforms simply cannot replicate in that environment.

    How do I govern Power Automate flows across a large organisation?

    Power Platform allows admins to apply environment-level DLP policies that control which connectors can access which types of data, and connectors can be blocked tenant-wide from the admin centre. Requiring solution-aware flows before anything reaches a production environment adds another layer of control, giving teams a clear audit trail when something needs investigating.

    What is a DLP policy in Power Platform and why does it matter for enterprise teams?

    A DLP (Data Loss Prevention) policy in Power Platform defines which connectors can interact with business or sensitive data within a given environment. For enterprise teams handling HR or finance data, these policies are a security requirement rather than an optional feature, and they are enforced at the tenant level rather than left to individual flow builders.

    When should I question a low-code platform comparison for enterprise use?

    Most platform comparison lists are designed for small teams starting from scratch with no existing infrastructure, which is a very different situation from a large organisation with an established Microsoft 365 tenancy and security requirements already in place. If a comparison does not address governance at scale, service principal authentication, or tenant boundary controls, it is not evaluating the same things your enterprise actually needs.

    This post was inspired by The 7 best low-code automation platforms in 2026 via Zapier Blog.

  • Your Copilot Studio Agent Passed Every Test and Still Failed in Production

    Your Copilot Studio Agent Passed Every Test and Still Failed in Production

    I came across a post from Zapier Blog about AI agent evaluation, and it described something I keep seeing inside large organisations: an agent that looks perfect in a demo, gets signed off, goes live, and then immediately starts doing things nobody expected. Wrong tool calls. Conversation loops that never resolve. Outputs that look confident and are completely wrong. The post frames this well as a sandbox problem. But the fix it describes, better test coverage and smarter metrics, only gets you partway there. The deeper issue with Copilot Studio agent testing is not the quantity of your tests. It is what you are actually testing for.

    Why Demo-Passing Agents Break in Real Workflows

    When a team builds an agent in Copilot Studio, they test it against the happy path. A user asks a clean question. The agent triggers the right topic or action. The response looks good. Someone in the review meeting says it works great. The agent gets promoted to production.

    The problem is that real users do not ask clean questions. They ask incomplete ones. They switch intent halfway through a conversation. They paste in text that includes formatting your prompt never anticipated. They use your agent for things it was never designed to do, because nothing in the interface tells them not to.

    None of that shows up in a demo. It shows up three days after go-live when someone forwards you a conversation log that reads like a stress test you forgot to run.

    The Three Failure Modes I Keep Seeing in Copilot Studio Agents

    After building and reviewing a number of agents internally, the failures cluster into three patterns.

    Topic misrouting at the edges. Your agent routes correctly when the user says exactly what you expected. But natural language is messy. When a user’s phrasing sits between two topics, the agent picks one confidently and gets it wrong. You only discover this when someone captures a failed session and traces it back. By then, a dozen other users have hit the same wall and just stopped using the agent.

    Action failures that degrade silently. A Power Automate flow or a connector action fails in the background and the agent carries on as if nothing happened. No error surfaced. No fallback triggered. The user gets a response that implies the task completed. It did not. This is the agent equivalent of a flow that retries quietly and masks the problem until the load goes up. I wrote about that pattern in the context of Power Automate throttling limits breaking flows under real load. The same logic applies here: silent success is not success.

    Prompt instruction drift under real data. Your system prompt was written against clean test data. Production data is not clean. It has unexpected characters, long strings, mixed languages, or values that push the model toward an interpretation you did not intend. The agent’s behaviour drifts. Not catastrophically. Just enough to become unreliable in ways that are hard to reproduce and harder to explain to stakeholders.

    How to Build a Behavioral Test Suite Instead of an Output Checklist

    Most teams build an output checklist. Did the agent return the right answer for these ten questions? That tells you almost nothing about production behaviour.

    What you actually need is a behavioral test suite. The difference is this: output testing checks what the agent said. Behavioral testing checks how the agent handled the situation.

    Here is how I approach it inside Copilot Studio before promoting anything to production.

    Build adversarial input sets, not just representative ones. For every topic your agent handles, write three versions of the trigger: the clean version, an ambiguous version that could belong to two topics, and a broken version with incomplete or oddly formatted input. If the agent routes all three correctly, you have something worth shipping. If it fails on the ambiguous case, you have a routing gap that will hit real users constantly.

    Test conversation state, not just single turns. Copilot Studio agents hold context across a conversation. Test what happens when a user changes their mind on turn three. Test what happens when they ask a follow-up that assumes context the agent should have retained but might not. Single-turn testing misses an entire class of failure that only appears in multi-turn sessions. This is also why agentic workflows require a fundamentally different design approach, not just an AI layer placed on top of existing processes.

    Inject real data samples into action inputs. Pull a sample of actual data from your environment and run it through the actions your agent calls. Do not use synthetic test data if you can avoid it. Real data has edge cases your synthetic data will never cover. If your agent calls a flow that queries a SharePoint list, run the query against the actual list with actual entries, including the ones with blank fields and formatting you did not anticipate.

    Define explicit fallback behaviour and test it deliberately. Every agent should have a defined behaviour for when it cannot complete a task. Most teams add a fallback topic and assume it works. Test it by constructing inputs that should trigger it. If the fallback does not fire, or fires on the wrong inputs, fix it before go-live. A graceful failure is far better than a confident wrong answer.

    What to Monitor After Go-Live and When to Pull an Agent Back

    Testing before launch is necessary but not sufficient. Agent behaviour shifts as the inputs it receives in production diverge from what you tested against. You need monitoring in place from day one.

    Track escalation rate and abandon rate per topic. If a topic is seeing significantly higher escalations than others, that is a signal of routing or response quality problems, not user error. Track action failure rates separately from conversation outcomes. An agent can complete a conversation and still have failed to do the thing the user needed.

    Set a threshold before launch. If escalation rate exceeds a number you agree on in advance, or if a specific action is failing more than a defined percentage of the time, you pull the agent back or disable the affected topic. The threshold is arbitrary. Having no threshold at all is not.

    The agents I have seen hold up in production are not the ones with the most sophisticated prompts. They are the ones where someone spent real time on the failure cases before launch and built actual monitoring into the plan from the start.

    If you are still signing off agents based on demo performance, you are not testing. You are hoping.

    Frequently Asked Questions

    Why does my Copilot Studio agent testing pass in demos but fail in production?

    Most Copilot Studio agent testing is built around ideal user inputs and predictable conversation paths, which do not reflect how real users actually behave. In production, users ask incomplete questions, switch intent mid-conversation, and use the agent in unintended ways that no demo ever surfaces. Testing needs to go beyond the happy path to catch these edge cases before go-live.

    What are the most common failure modes in Copilot Studio agents?

    The three patterns that appear most often are topic misrouting when user phrasing falls between two intents, action failures that complete silently without triggering any error or fallback, and prompt instructions that break down when they encounter messy real-world data. Each of these can go undetected in testing because they only emerge under realistic conditions.

    How do I know if a Power Automate action failed inside my Copilot Studio agent?

    Silent action failures are a serious risk because the agent can continue the conversation and imply a task completed when it did not. You need explicit error handling and fallback logic in your flows so that failures surface to the user rather than being masked by a confident-sounding response.

    When should I test my Copilot Studio agent against real production data?

    You should test against realistic data before promotion to production, not after. System prompts written against clean test data can behave unpredictably when they encounter unexpected characters, mixed languages, or long strings that only appear in live environments. Incorporating a sample of real or representative data into your test suite is a necessary step before sign-off.

    This post was inspired by AI agent evaluation: How to test and improve your AI agents via Zapier Blog.