RAG vs Fine-Tuning for Enterprise LLMs: Which to Choose?

Key takeaways

• RAG is where most enterprise use cases start. At the device maker in this guide, field engineers need this month's safety notices, filtered by region, and retrieval is the only way to give them that.
• Fine-tuning comes in when behavior misses a target. The device maker's complaint coder picked the right code 89% of the time with prompting alone, and the quality team needs 97%.
• Permissions go into the retrieval query. Filtering afterward means restricted text has already passed through the pipeline and its logs.
• Personal data stays out of training sets. Once someone's details are in the weights, a deletion request turns into a retraining cycle.
• The data layer sets the schedule. The device maker fails three of seven readiness checks, so its first few weeks go to permissions, tagging and an evaluation set before anyone picks a model.

We've been building on AWS since 2008, and a good share of that work has been audit-heavy AI systems for healthcare, financial services and manufacturing teams. In those programs, the fine-tuning vs RAG question has used up more meeting time than almost any other architecture call, while the thing that actually decides the outcome sits on the sidelines. That's the data layer: where your knowledge lives, who is allowed to see it, how current it is, and whether you can prove all of that to an auditor. It goes a long way toward explaining why so few programs pay off. In McKinsey's 2026 State of AI survey, 37% of respondents said their use of AI had produced at least some EBIT impact, about where the number sat a year earlier.

To keep this concrete, we'll follow a made-up but typical US medical device maker. It makes infusion pumps and patient monitors, brings in about $1.4 billion a year, and sells in Europe through a subsidiary in the Netherlands. Two LLM projects are on its list. The first is a service assistant for about 700 field engineers, who today dig through service manuals, firmware release notes and field safety notices stored in SharePoint, some of them restricted by region or customer contract. The second is complaint coding. The quality team codes about 3,000 complaints a month from Salesforce against the IMDRF adverse event terminology that FDA's reporting codes map to. Last year the company tried to do both with one model fine-tuned on a dump of manuals and old tickets, and we'll come back to why that went sideways.

What Is the Difference Between Fine-Tuning and RAG?

The device maker learned the difference from last year's pilot. A month after a field safety notice changed the battery replacement steps on one pump model, the pilot was still giving technicians the old ones. It had been fine-tuned on manuals exported in January, so it answered from what it had been trained on.

A RAG system would have pulled the notice itself and cited it. Retrieval decides what the model knows at the moment someone asks, and fine-tuning shapes how it behaves on every call. The pilot's tuning had done its part well. Its answers were fluent and laid out the way technicians liked, and they were a month out of date. From then on, the team's questions were about data rather than models.

Put to the device maker's own questions, the three options compare like this:

 

When a steering committee wants one rule, we give them the line at the top of our decision flow: retrieve for knowledge, fine-tune for behavior. At the device maker, that rule sends the field safety notices, the regional restrictions and anything a customer could ask to delete into retrieval. The only thing left as a candidate for training is the shape of the complaint output.

When Does Fine-Tuning an Enterprise LLM Earn Its Keep?

Less often than vendors suggest, but complaint coding at the device maker is one of the cases where it can. It needs a strict output format that prompting hasn't been able to hold, and it uses a vocabulary the base model keeps misreading. The other two cases where we'd consider it are volume high enough that a smaller tuned model changes the economics, and customer-facing agents that have to stick to a regulated script.

Every complaint has to come out as a device problem code, a component code and a short rationale, in the JSON the complaint system imports, and the vocabulary is IMDRF's, not plain English. The quality team started where we'd always start, with a well-prompted base model that pulls the code definitions through retrieval.

On a 500-complaint evaluation set built from audited records, it put the right device problem code first 89% of the time, against a 97% target, and 6% of its outputs broke the import schema. At 3,000 complaints a month, that's about 180 records a month someone has to fix by hand before review even starts. That measured miss is what earns a fine-tuning experiment.

Recent research backs up the split. In a January 2026 study of multi-hop question answering, the authors found that unsupervised fine-tuning gave only limited gains over base models, retrieval produced substantial and consistent improvements on new knowledge, and supervised fine-tuning reached the highest overall accuracy. Put simply, training on raw documents is a poor way to teach a model facts, and training on well-built examples is a good way to teach it a skill. Last year's pilot at the device maker did the first, on raw manuals.

On AWS, the options line up like this. Amazon Bedrock currently lists three customization methods: supervised fine-tuning, reinforcement fine-tuning and distillation. For continued pretraining on a large unlabeled corpus, we'd use Amazon SageMaker AI.

 

We'd put one caution in bold on any RFP. Content that changes weekly, carries per-user permissions or holds personal data someone can ask you to delete doesn't belong in a training set. At the device maker, that rules out the safety notices, the regional bulletins and the patient details in complaints. Training on that kind of content creates obligations you pay for at audit time.

Alt text: Decision flow for choosing fine-tuning vs RAG for an enterprise LLM, from data-layer readiness to production

At the device maker, the service assistant leaves the second decision toward RAG. Complaint coding goes down to the third, misses on format and accuracy, and lands in the fine-tune box as a hybrid, since the code definitions still come through retrieval. Neither one gets past the first decision yet, though, which is where the next section starts.

Why Is the Data Layer the Real Decision Behind Every Enterprise LLM?

Because both approaches inherit whatever is wrong with the data underneath. A retrieval system reading stale, unpermissioned files will hand the wrong policy to the wrong person and sound sure about it. A model trained on unlabeled exports picks up the inconsistencies along with everything else. In the stalled programs we've been asked to look at, the cause was the data far more often than the model choice.

Gautham B, one of our forward-deployed engineers, put it plainly in his analysis of agentic RAG in production:

"When a metric slips, the cause is more often the data than the model." He followed it with the design rule we apply on every regulated build: "The rule we'd follow is that the user's permissions go into the retrieval query, not onto the results."

That second line matters more than it looks. If you filter results after retrieval, sensitive chunks have already passed through the pipeline, sat in logs and possibly landed in a prompt cache. At the device maker, that's how an EU-only bulletin would reach a technician in Ohio. Push the entitlements into the query, and restricted content never moves at all.

For the device maker, AWS can carry part of the load. Its SharePoint library could feed an Amazon Bedrock Managed Knowledge Base, generally available since June 2026, which brings each document's permissions along when it ingests it and filters what a given engineer gets back at query time.

AWS is clear that this isn't a security boundary. The application still has to authenticate the engineer and pass a verified identity, and that identity has to match the email address each source system holds. A lot of first-phase work goes into making those identities line up.

How ready is your data layer? A seven-point scorecard

Run this with your data and security leads before anyone debates models. It takes about an hour, and it will tell you where the first eight weeks of budget should go. It's the same argument behind our view on why most enterprises will fail at agentic AI: the source data decides the outcome long before the model does.

The device maker passes four. Every source is named and owned, field safety notices have an agreed sync interval, SharePoint versions every document, and the Dutch subsidiary knows which records have to stay in the EU. It fails the other three. SharePoint access groups don't map cleanly to regions or customer contracts, nobody tags patient details in complaint narratives before they're copied around, and the service assistant has no evaluation set anyone owns. Three fails puts it in the yellow box, so the first weeks go to entitlements, tagging and a set of real technician questions with approved answers. The fine-tuning vs RAG choice can wait.

 

What Does a Production Data Layer for Enterprise LLMs Look Like on AWS?

The AWS pieces are mature, and wiring them together is where the work is: authorization, lineage and evaluation have to hold up under production load and an audit.

Gautham made the same point about the platform in his breakdown of Amazon Bedrock in production: "Amazon Bedrock resolves about a fifth of the decisions between a prototype and production." On governance he was blunter. "Guardrails cover the language layer only," he wrote, and permission to act "belongs in the tool implementation and the IAM role the agent assumes."

At the device maker, one governed data layer would feed both projects. The service assistant reads from the index, and complaint coding's training examples come out of the same classified, permissioned, lineage-tracked data. Here's what each layer would hold:

Which vector store should an enterprise RAG architecture use?

There's no single right answer, and the choice is easier to reverse than most teams fear if lineage is in place. Here's how we'd call it:

 

Hybrid search earns its keep when exact terms matter, which is common in service work. In Hibu's conversational analytics build, a purpose-built MCP connector gives Claude hybrid keyword and semantic search across Hibu's OpenSearch index, and the system analyzes 1.82 million inbound lead calls a month. For the archive end of the spectrum, AWS says S3 Vectors returns infrequent queries in under one second and can cut the cost of storing and querying vectors by up to 90%.

The device maker's knowledge at least lives in two systems, SharePoint and Salesforce, so its gaps are about permissions and tagging rather than location. A company with records scattered across on-premises warehouses and SaaS exports has a bigger job first, and there we'd begin with AWS data lake consolidation. Nothing gets embedded until lineage and access controls exist.

How Do Compliance Requirements Change the Fine-Tuning vs RAG Decision?

Mostly, they push sensitive knowledge toward retrieval. At the device maker's next audit, an auditor might pull one answer the assistant gave a technician in Lyon and ask three things. Who else could have seen that bulletin? Which document did the answer come from? And if the hospital asked, could the record be removed? A RAG system with a proper data layer can show all three. A fine-tuned model can show some of the first two and very little of the third.

What is the erasure problem with fine-tuned models?

Here's how that plays out. A biomedical engineer at a hospital in Rotterdam asks the Dutch subsidiary to erase his personal data under GDPR. His name, email and phone number are in years of service tickets. With RAG, the team deletes the source records, re-indexes and logs it. Had last year's pilot been trained on those tickets, his details could be sitting in the weights, and nobody has a reliable way yet to pull one person's data back out of a model. Regulators aren't going to wave this through. In its December 2024 opinion, the European Data Protection Board said models trained on personal data can't always be considered anonymous. The only clean fix is to retrain on data with his tickets removed, so one deletion request turns into a model release.

We'd avoid that by keeping personal and regulated data out of training sets from the start. The complaint coder can learn its format from de-identified or synthetic examples while retrieval brings in sensitive facts under access control. Complaint records work differently, since a device maker has to keep them, but the patient details inside them still don't belong in a training set. For complaint coding, names, dates of birth and hospital details get masked before any example reaches a training job.

Which frameworks shape an enterprise LLM architecture?

Two rows matter most at the device maker. The Dutch subsidiary's tickets and complaints carry contact and health data, which puts GDPR first. And complaint files are records an FDA inspector will read under the Quality Management System Regulation, which took effect on February 2, 2026, so every coding decision needs a trail. The other rows are what most enterprises run into, each written as the question you're likely to hear.

 

The EU AI Act row deserves a note for European buyers. The Digital Omnibus on AI moved most high-risk obligations to December 2, 2027, for Annex III uses and August 2, 2028, for AI in regulated products. For the device maker, the 2028 date is the one to watch for AI built into its pumps, since neither internal tool here is a medical device. Teams that build logging and data governance into the data layer now won't need a second program to retrofit them later.

What does a governance gap actually cost?

The financial exposure is concrete. IBM's 2026 Cost of a Data Breach research put the global average breach at a record $4.99 million, up 12% on the year before, and found that among organizations reporting an AI-related security incident, 92% lacked proper AI access controls. That's exactly the gap permission-aware retrieval closes. Designing for the audit from the first week pays off, too. The HIPAA-compliant agentic claims platform we shipped for KlearTrust in twelve weeks passed HIPAA and internal audit on first review.

What Does an Enterprise LLM Really Cost Over Three Years?

Usually more than the token estimate. Over three years, the money goes to data preparation, evaluation and keeping the system current, and choosing fine-tuning or RAG mostly moves spend between those lines rather than changing the total. Cost is already holding teams back: about one in five respondents in McKinsey's survey said AI operating costs, tokens included, had constrained their use of AI.

The chart below compares the two over three years. It uses relative weights instead of prices, since unit prices change every quarter and the shape of the spend doesn't.

At the device maker, the two projects land on opposite sides of that chart. The service assistant's money goes to data preparation, vector storage and evaluation. Complaint coding adds training runs on top, and the bigger cost there is keeping the tuned model current.

What does the base-model treadmill do to the budget?

New foundation models arrive every few months, often cheaper or more capable than the last. A RAG system can usually switch after an evaluation pass. A tuned model can't, because it's tied to its base, so each upgrade means curating, training and validating again. Bedrock's model lifecycle policy puts dates on that. Most models get a six-month Legacy period before end of life, and once a base model is in Legacy, new fine-tuning jobs on it aren't allowed. For complaint coding, we'd budget a retrain every time its base model is retired. Over three years, that refresh tax can easily exceed the original training bill. When we model total cost of ownership with a CFO, it's the line that changes minds.

Which levers bring enterprise LLM costs down?

Routing comes first. Simple lookups can go down a lean retrieval path, with multi-step reasoning saved for the questions that need it. In the same agentic RAG analysis, Gautham found that "Sending simple questions down the standard path cuts the bill by about 40 percent." Prompt caching helps with the repetitive part of each request, such as system instructions and stable context.

The purchasing mode needs settling before the architecture hardens. On Amazon Bedrock, inference profiles don't support Provisioned Throughput, so choosing one shapes your resilience design. Distillation should wait until volume is proven. We'd tune a smaller model only after the RAG version has shown which behaviors matter at scale.

Much of the budget may already exist, too. Our case for funding AI from your existing budget comes down to two lines: "Most of the AI budget you think you need is already in the building. It is paying interest on the old stack." The database license up for renewal, the warehouse nobody runs new workloads on and the integration tier billing hours to keep itself alive can all be reallocated. That turns the LLM program into a capital allocation decision instead of a new line item for the board.

How Should You Sequence an Enterprise LLM Implementation?

We'd prove the hardest part first, which is usually permissions or retrieval quality, and then scale one signed gate at a time. The programs we've seen go sideways tried to pick a model, build the interface and negotiate data access all at once.

At the device maker, phase two would prove permission-aware retrieval on one product line's SharePoint library, EU-only bulletins included, before anything else gets built. Complaint coding would follow as its own slice, starting from the prompt baseline, and a tuned model would come in only if the side-by-side report in phase four showed the baseline still short of its target.

For a scoped use case at moderate complexity, this runs roughly 11 to 20 weeks from scoping to handoff, and our median across engagements is 12 weeks to production. Final ranges belong in the statement of work, and we'd be wary of anyone who quotes a date before phase one. Each gate closes with a customer sign-off, which is how we work on every engagement. On generative AI consulting work, the date and the fee are both fixed before the build starts.

What Are the Warning Signs an Enterprise LLM Program Will Stall?

Most of them showed up in last year's pilot at the device maker, which makes it a useful checklist.

The pilot was one model fine-tuned on an export of service manuals and two years of service tickets. It looked good in the demo. Within a month, a field safety notice changed a battery replacement step, and the model kept giving the old one, because nothing short of retraining could update it. A technician in Ohio got an answer drawn from a bulletin meant only for EU customers. Nobody could say which document an answer had come from, so quality wouldn't sign off. Nobody had built an evaluation set, so the debate about whether it was good enough ran on anecdotes. Nobody had priced an answer, either. The end came when it suggested the wrong code on a complaint. Nobody knew whether to roll back, retrain or switch it off, an ad hoc meeting was called, and the pilot was shelved.

The fixes are mostly unglamorous. This time, each SharePoint library gets a freshness SLA, and the evaluation set is rescored every night, so a changed battery step shows up as a failed test within a day. What the assistant may do without a person is written into tool code and IAM roles before the build. Cost per answered question is tracked from the first slice. The service team now owns 150 real technician questions with approved answers. The rollback plan gets written in phase two, and every source system has its access tested during scoping.