Plenty of companies now have an agent pilot that impressed somebody senior. Ask how many of those pilots are allowed to change a record in a production system without a person clicking approve, though, and the number drops to almost nothing. The analysts are seeing the same thing.
Gartner's forecast is that more than 40% of agentic AI projects will be scrapped by the end of 2027, citing "escalating costs, unclear business value or inadequate risk controls," and when McKinsey surveyed 1,719 people for its 2026 State of AI report, only "about two in ten respondents report reaching the scaling phase across their organization with AI agents."
We wrote this for engineering, data and product leaders who already have a pilot or two running and want a straight answer on what production will ask of them. Our enterprise AI strategy framework calls this the execution layer, where models, orchestration and tools live, and in our experience it's the layer that gets rushed so the demo can happen on time.
What follows is the five capabilities we want before an agent gets real authority, the architecture and AWS services we'd use, a comparison of delivery approaches, and the governance, operations and rollout habits that keep shadow AI from filling the gap.
Mostly the things nobody asked about during the demo. Take a typical refund agent piloted by a support team: it read the ticket, checked the order in the commerce platform and drafted a refund, and the demo went great. Then security asked to see the exact prompt, model version and order data behind one refund it issued on a Tuesday in May, and there was no record of any of it, because the pilot only ever had to prove that a model could do the job.
That refund agent had the same gaps we find under most pilots. Its system prompt lived in a notebook, so when it started approving refunds a little more generously one week, nobody could tie the change to a commit. Its quality had been judged by whoever ran the demo, so the model upgrade in June went out with no regression test.
It read orders through a service account that could also see customer payment details, which the task never needed. Its logs kept the final reply but none of the lookups behind it. And nothing capped its spend, so a messy ticket that took nine model calls instead of two simply cost more and nobody noticed.
Each of those is cheap to design in and expensive to bolt on later, which is why we'd run production as its own workstream from the first sprint, not as a phase that starts once the demo goes well.
When that agent finally did quote a customer the wrong refund amount, five teams got pulled into the thread: whoever wrote the prompt, the platform team, the data team that owned the order tables, the person who picked the model and the security folks who wrote the policy. Each of them could point at somebody else's layer with a fair argument.
Ordinary software mostly avoids this because ownership follows the code. With agents we'd name one person as owner of the agent's behavior before anything ships, and give the other teams a defined say in that person's decisions.
The five we check are reliability, governance, integration, observability and operations. An agent can run without one of them, sometimes for months, but we wouldn't give it real authority over a system of record until all five are in place.
Reliability is what lets the refund agent behave on Black Friday the way it did in testing, after a model upgrade and with customers typing things no test set anticipated. We'd get there with typed tool contracts checked before anything executes, retries and timeouts with a fallback around each model and tool call, a hard cap on the number of steps a task can take (with a handoff to a person when it's hit), and regression evals that run whenever anyone changes a prompt, a model or a tool.
For AI agent governance we'd want three answers written down for every agent: what it's allowed to do, whose authority it's acting on, and how you'd prove both after the fact. The NIST AI Risk Management Framework and ISO/IEC 42001 are useful for the vocabulary and the management system around all this, but the controls that stop the refund agent issuing a $4,800 credit are much smaller things, such as a permission on that one action, an approval threshold above a dollar amount, version control on the prompt and policy, and an audit record of every task.
IBM's 2025 breach research shows what happens without them. Of the 600 organizations studied, 63% had no AI governance policy at all, and among those that reported an AI-related breach, 97% "lacked proper AI access controls."
An agent that can't reach your systems of record is a nice chat window and not much more. We'd route every tool call through a gateway that checks identity, scopes and rate limits, so the agent never holds credentials of its own. The usual blocker is the legacy system with no clean API, which is why we'd often run an agent program next to application and database modernization instead of waiting for that work to wrap up.
Good AI agent observability means that six weeks after a complaint, you can still replay exactly what the agent saw and did, including the input, the retrieved context, each model call and its prompt version, every tool call, the tokens, the latency and the final action. We'd use the OpenTelemetry semantic conventions for generative AI for those spans, mainly so agent traces end up in the same backend as everything else you run instead of in one more dashboard nobody checks.
Operations covers the unglamorous daily work of keeping agents running, from cost per task and latency targets to incident response, model deprecations and release notes for prompt changes. In our experience it's also the capability most likely to be left out of the budget, because it looks like overhead until the first incident, and then it suddenly looks cheap.
If you're building on AWS, most of this plumbing now ships as part of Amazon Bedrock AgentCore, which became generally available in October 2025. We'd still design the controls ourselves, but there's a lot less to build from scratch:
AgentCore Policy and Evaluations only reached general availability in March 2026, so if your last look at AgentCore predates that, it's worth another one.
Before we'd call an agent production-ready, we'd want to tick every line of this list:
The agentic AI architecture we'd recommend splits a production agent into layers that each have one job, mainly so a change in one doesn't force a rewrite of the others. If the refund agent moves to a newer model next quarter, its tool definitions shouldn't have to change, and if it later needs to read from the returns warehouse as well as the order system, nobody should have to touch the guardrails. The diagram below runs from top to bottom, and the evals and guardrails sit across all of it rather than in one box.
This layer puts one or more LLM providers behind a single interface and routes each task to the smallest model that does it well enough. For the refund agent, sorting the ticket might go to a small, cheap model while drafting the reply to the customer goes to a bigger one. We'd keep model IDs in configuration instead of code, so when a provider retires a model the fix is a config change plus an eval run, not a release. On AWS that's usually Amazon Bedrock, and outside it a direct provider API or an open-weight model served with something like vLLM does the same job.
The runtime keeps track of where a task is, which means the plan, the state between steps, short-term memory and the step limit. Most teams end up with a graph-based framework like LangGraph or Strands Agents running on a managed service such as AgentCore Runtime. For a narrow, high-volume job where you want every transition written out by hand, a small in-house state machine is still a perfectly good choice.
Tools are described with typed schemas, and more of them are exposed over the Model Context Protocol, which has been looked after by the Linux Foundation's Agentic AI Foundation since December 2025. The gateway is where the checks happen. When the refund agent asks to issue a credit, the gateway confirms who it's acting for, whether that action is on its allowlist and whether it's still under its rate limit, and only then does the call go through.
When the refund agent got something wrong in testing, the cause was usually something it had retrieved, such as last year's returns policy or an order from the wrong region. So we'd have it search curated indexes that carry their permissions along with the content, rather than a folder of exported PDFs, and check access at the moment of the query using the entitlements of whoever asked, whether that's the customer or the support rep.
This is also where data modernization work pays off for an agent program. When lineage, classification and access rules already live in the data layer, retrieval can just inherit them, and each agent team doesn't have to rebuild them. For the stack itself, vector search in OpenSearch or pgvector, a feature store for structured context and a lakehouse catalog for lineage cover most cases.
Each call should carry the identity of the person who asked, not only the agent's, so the agent can't see or do anything that person couldn't. On AWS, AgentCore Identity does this with a token vault for OAuth tokens and API keys, support for both machine-to-machine and user-delegated OAuth, and built-in connectors for services like Salesforce, Slack and Jira. Whichever tool you use, we'd keep secrets out of agent code and config files entirely.
We'd put LLM guardrails on both sides of the model. A customer message gets screened for prompt injection and card numbers before the model sees it, the drafted reply gets checked against policy before the customer sees it, and any tool call that changes something gets checked before it runs.
For the threat model we'd start with the OWASP Top 10 for LLM Applications, because it's still the clearest list for prompt injection and excessive agency. Then we'd add the OWASP Top 10 for Agentic Applications, which came out in December 2025 and covers problems you only get with agents, like tool misuse, identity and privilege abuse, and one agent's failure cascading into another's.
OWASP calls its central principle "least agency," and in practice that means the refund agent gets the tools and permissions for refunds and nothing else.
The limits themselves need to live outside the prompt. Suppose the refund agent is told in its system prompt never to credit more than $500. Then a customer writes in, angry about a $4,800 sofa that arrived with a split frame, and three messages later the model has decided this case is the exception. With the $500 rule in the gateway instead, set up in AgentCore Policy or in a policy engine you run yourself, the credit call fails, and the ticket lands with a person who can make the call on the sofa.
This is the layer that tells you whether the other six are doing their jobs. We'd run offline AI agent evaluation suites before every release, with the power to block it, score a sample of live refund tickets every day, and send the traces from both to the debugging view your engineers already use and to the monthly cost report finance keeps asking for.
For the first agent, the path matters less than you'd think. It starts to matter when the second request lands, say from the collections team wanting an agent for payment reminders after they've seen the refund agent work. If each team built its own stack, collections starts from zero on guardrails, evals and tracing, and probably repeats a few of the refund team's mistakes along the way.
On a shared platform, the collections engineers would inherit all of that and could spend their first month on the reminder logic itself. Here's how the three common setups compare from the point of view of that second agent:
The third row is the one we'd spend the most time on with a leadership team. Suppose the collections agent went live without tracing and a complaint forced the team to add it. They'd have to rework the prompts so the guardrails fit, add identity plumbing to each tool and put traces into code that was never written for them, all inside one sprint while the agent is still taking traffic.
A shared platform does cost more before the first agent ships, but the bill for the second one is much smaller, which is why we build the platform baseline and the first agent in the same engagement when we deliver production AI agents. Our point of view on agentic AI goes into the five architecture decisions behind that path.
How Do You Run Enterprise AI Agents in Production?
Once the refund agent is live, it will meet tickets nobody tested and model updates nobody scheduled, so we'd want three things settled before launch. Those are an architecture pattern that suits the work, controls written in code where nobody can quietly skip them, and a runbook somebody wrote ahead of time rather than during the first incident.
Three patterns cover most enterprise use cases:
Whichever one you choose, we'd keep the number of tools per agent low. An agent holding forty tools takes longer to plan, is harder to write evals for and has more ways to overreach than three agents with a dozen tools each, and splitting it up is usually less work than trying to test it as one.
For the refund agent, the controls we'd put in place aren't exciting, and that's mostly the point. Nobody edits its prompt in a console. The prompt, the tool schemas and the guardrail settings are kept in Git with the rest of the application, so changing any of them means opening a pull request that someone reviews, and what ships is a tagged release. We'd also keep a golden set of real refund tickets, somewhere between 200 and 500 of them, and run the agent through it before every release. It gets scored on whether it got the answer right, whether that answer was grounded in the order data, whether it followed policy and whether it called the right tools, and a drop below the threshold on any of those holds the release.
We'd also sort every action the agent can take into three tiers. Looking up an order is a read, so it runs on its own. Adding a note to the ticket is a write, but someone can delete the note, so it runs on its own and a reviewer looks at a sample each week. Sending money back to a card is the one action nobody can take back, so every refund waits for a support lead to approve it until the eval results justify letting smaller refunds through on their own. We'd put the $500 limit, the data residency rules and the entitlement checks in the gateway as well as the prompt. We'd also tie each of these controls to a risk named in the NIST Generative AI Profile (AI 600-1), which saves a lot of back and forth when the audit team shows up with its questionnaire.
Regulated work uses the same design with stricter settings. When KlearTrust put an agentic claims platform into production in twelve weeks, it relied on the same kind of audit trail and action tiers, and the platform "passed HIPAA and internal audit on first review."
Before launch, we'd want five things written down. The first is cost, which we'd track per completed refund instead of per model call, with an alert when spending drifts away from the agent's own baseline, because a planning loop can triple the calls on a messy ticket. The second is latency. We'd set p50 and p95 targets for each kind of task, and for a customer waiting in a chat window we'd aim to keep the p95 under eight seconds, closer to five if we can, while a back-office agent can wait longer and use batch pricing.
The third is incidents. Severity levels for wrong, harmful and runaway behavior should be agreed in advance, and every agent needs a kill switch that stops its tool calls while still letting it answer read-only questions. The fourth is the model lifecycle, which means a deprecation calendar for every model in use and an eval run against the replacement at least one release cycle before cutover. The fifth is change management, so when the refund agent's prompt or policy changes, the support managers who rely on it get release notes the same way they would for an API change.
We'd also check the list against the AWS Well-Architected Generative AI Lens, mostly its operational excellence and cost optimization pillars. AWS updated it in November 2025 with new guidance on agentic AI, so older notes on it are out of date.
By treating the first agent as the template for the next ones. In practice that depends on who's on the team, how often the agent's behavior gets reviewed and the order in which it's rolled out, much more than on any one engineer being brilliant.
For the refund agent, we'd want five names on the roster. One is the agent product owner, who in this case would probably be the head of support operations, since that's the person who hears about it when refunds go wrong. Two or three forward-deployed engineers write and maintain the prompts, tools and evals for refunds specifically. Someone from the platform team looks after the gateway, the runtime, observability and the guardrail services, which every other agent will share too. A data engineer keeps the order and policy indexes current and the access rules correct. The fifth is a risk partner who's involved from the first week and signs off on the action tiers as they're set, rather than reviewing everything at the end and sending it back.
The repetitive work, such as generating tool definitions from the commerce platform's API spec, turning last year's tickets into eval cases and wiring up traces, can mostly be automated. Where the agent's authority stops is a judgment call, though, and a person should make it.
Every week, we'd have the refund agent's owner and engineers look at eval trends, a sample of traces and any incidents. Monthly, the risk partner sits in and uses the same evidence to decide on tier changes, for example whether refunds under $100 have earned the right to go out with sampled review instead of an approval each time. At the quarterly review, leadership sees every agent side by side and picks which ones get more scope, which keep what they have and which get switched off. We'd tie these reviews to phase gates with signed acceptance, so each stage has to come with evidence rather than a demo, and people show up having read it.
Sequencing is also how you avoid shadow AI. When the sanctioned path is slow, teams route around it with personal API keys, and IBM's 2025 breach data put a price on that: organizations with high levels of shadow AI paid an extra USD 670,000 per breach on average. We'd roll out in four stages, each with an exit criterion:
Shadow AI also shrinks when the official route is quicker than the unofficial one. A team that gets pre-approved model access, a tool gateway and an eval template on its first day has little reason to put a model API key on somebody's corporate card.
We're happy to look at a pilot that's stuck. Book 30 minutes with the engineer who'd own the work, bring the use case and a rough sketch of how it runs today, and you'll leave knowing which of the five capabilities it's missing. When there's a fit, we follow up with a scoped, fixed-fee proposal.