AWS Data Pipeline Modernization: Architecture, Cost, Cutover

Key takeaways

• Most pipeline estates are past due before anyone says so. The home goods retailer in this guide fails all five readiness signals, starting with a nightly batch that now finishes after 6 a.m.
• Modernization changes how each layer behaves, not just where it runs. Moving the retailer's 600 jobs onto bigger servers would keep the 6 a.m. finish and the access tickets.
• AWS Data Pipeline jobs need a new home. AWS closed the service to new customers in July 2024, and the retailer's DynamoDB exports can give way to a zero-ETL integration with Amazon Redshift.
• Compliance controls belong inside the pipeline. Card data gets tokenized before it lands, and snapshots on tables with EU personal data expire well inside GDPR's one-month response window.
• The license renewal sets the date. The retailer's ETL suite renews in June 2027, so cutover and decommissioning are planned backward from it.

If your nightly batch wrapped at 6 a.m. and nobody trusted the revenue dashboard until noon, you already know where the AI budget is leaking. The pipelines most mid-market and enterprise companies run today were built to feed weekly reports. They're now being asked to feed pricing models, fraud scoring and customer-facing agents, and the seams show up first in the data. In a 2026 Harvard Business Review Analytic Services survey sponsored by Cloudera, only 7% of respondents said their organization's data was completely ready for AI.

We've been building on AWS since 2008, and much of that work has been data platforms for regulated companies, where one audit finding can stall a program for a quarter. This guide is for the CIO, CDO, CTO and CFO who have to sign the modernization plan and live with the result.

To keep it concrete, we'll follow a made-up but typical company: a US home goods retailer with about $1.2 billion in sales, 160 stores, an online store and a fulfillment center in the Netherlands that ships to EU customers. Its data runs through about 600 nightly jobs in an on-premises ETL suite, a SQL Server warehouse full of stored procedures, cron scripts on two shared virtual machines, and a dozen AWS Data Pipeline jobs the e-commerce team built in 2021 to export DynamoDB order tables to Amazon S3. The ETL license, about $450,000 a year, renews in June 2027. Merchandising wants hourly inventory for a markdown model, the fraud team wants order signals within minutes, and a customer-service agent is on the 2027 roadmap.

What does data pipeline modernization on AWS actually involve?

It means moving ingestion, transformation, orchestration and serving off legacy ETL servers, cron jobs and on-premises warehouses onto AWS managed services, with quality checks, lineage and access controls built into the pipeline itself. The result is current, governed data that analysts and AI agents can query through a supported path.

Moving the same jobs onto bigger servers is a hosting change. If the retailer did only that, its batch would still finish after 6 a.m. and access would still run on tickets. Modernization changes how each layer behaves:

It's also a question of where the money comes from. The retailer's $450,000 ETL renewal, the support contract on its warehouse servers and the hours its engineers spend restarting failed jobs could all go to the AI program the board keeps asking about. That's the idea behind funding AI from your existing budget: "Most of the AI budget you think you need is already in the building. It is paying interest on the old stack."

How do you know your data pipelines are due for modernization?

When keeping them alive costs more than replacing them, counted in license spend, engineer hours and decisions made on stale data. We check five signals in every assessment, and each one can be measured with data you already have.

The retailer fails all five. Over two years, its median batch finish slid from 3:40 a.m. to 6:10 a.m., and the worst night ran past 9. Store managers spotted missing sales on the dashboard twice last quarter before the data team did. Changes to the e-commerce platform's API broke loads seven times in the same quarter. A new analyst waited nine business days for access to the sales tables, and the ETL renewal is eight months away. With three or more in the red, the decision is already made, and what's left is sequencing.

The architecture diagram is rarely what sinks these programs. Ammar Nizami, a senior forward-deployed engineer in our data engineering practice, makes the point in his analysis of why data modernization programs stall: "Most data modernization efforts do not fail at the architecture diagram." They fail months in, when nobody can say who owns a dataset, access grants have piled up for years, and the catalog lags what's actually running.

The retailer's own gut check went badly on all three. Nobody could name an owner for 11 of its 20 most-used datasets, nobody could show who had read the payments table last quarter, and when a store feed stopped one Sunday, the first person to notice was a regional manager on Monday morning. Each of those gaps belongs in the modernization scope, right next to the compute and storage work.

 

What does a modern AWS data pipeline architecture look like?

Four stages with two planes across them. Data comes from source systems, arrives through change data capture and streams, lands in an open-format lake on Amazon S3 where AWS Glue or Amazon EMR refines it, and is served through Amazon Redshift and Amazon Athena. Orchestration and observability run across all four stages, and so does governance, so a failed job or an unauthorized read is caught where it happens.

The diagram below is the reference pattern our data platform modernization work starts from. Every client ends up with a variation, because source systems, latency targets and regulators differ, but the layers hold.

For the retailer, AWS DMS would stream changes out of SQL Server instead of the nightly full extracts. Orders from the online store would arrive through Amazon Kinesis Data Streams, so the fraud team sees them in minutes, while point-of-sale files would keep landing in S3 every night, since nobody needs them sooner. Everything would end up in Iceberg tables, with Redshift serving finance and merchandising.

Which AWS services fit which pipeline job?

Service selection is where most cost overruns start. Teams pick the tool they know, then pay for that choice on every run. Here's how we match services to jobs:

 

Should you choose ETL or ELT, batch or streaming?

Choose by how fresh each decision needs its data, and expect to run both. ELT inside Amazon Redshift or on the lake suits analytics teams who model in SQL. Classic ETL in AWS Glue or Amazon EMR still wins when data has to be masked, tokenized or reshaped before it lands anywhere regulated, which is how the retailer's card data should be handled. Streaming earns its cost where minutes matter, like the fraud team's order signals. Batch is still right for the finance close and most reporting. Running both on one governed lake is normal, and it's how we design by default.

What replaces AWS Data Pipeline?

AWS stopped taking new customers for AWS Data Pipeline on July 25, 2024. Existing customers can keep running it, but AWS has said it doesn't plan new features, and it points workloads to the three services in the figure: AWS Glue, AWS Step Functions and Amazon MWAA.

The retailer's dozen DynamoDB exports are a good test. They feed one nightly sales reconciliation in Redshift, so a zero-ETL integration from DynamoDB into Redshift would replace them outright. Its cron-driven SQL jobs would move to MWAA, and since Amazon MWAA Serverless arrived in November 2025, a small team can pay only for the compute its tasks use instead of running an Airflow environment around the clock. Whatever the target, we'd use the move to fix scheduling, retries and alerting rather than copy old job definitions one for one.

Where do Apache Iceberg and zero-ETL fit?

Apache Iceberg gives the lake transactional tables, schema evolution and time travel on open files in S3, so Glue, EMR, Athena, Redshift and outside engines can all read the same data. Ammar's piece on Apache Iceberg on AWS puts the strategic side plainly: "Picking a table format is really picking who controls your data." He also warns that "The format is free, but maintenance isn't." Compaction and snapshot expiry belong in the operating plan, or with Amazon S3 Tables, which AWS launched in December 2024 to run that maintenance for you.

Zero-ETL integrations copy data into Amazon Redshift without pipelines you build or maintain. AWS's list of sources now runs well past Aurora, RDS and DynamoDB to self-managed Oracle, SQL Server, MySQL and PostgreSQL databases and SaaS apps such as Salesforce, SAP and ServiceNow. They suit operational reporting. They don't replace transformation logic, quality rules or masking, so we'd treat them as one ingestion option inside the wider design.

How does a modern pipeline get your data ready for AI agents?

AI agents read data differently from dashboards. They send many small, unpredictable queries, they need to know what a field means, and every answer has to trace back to a source. Ammar's piece on the data lakehouse for agentic AI opens with that contrast: "Agents query nothing like dashboards do." It tightens freshness targets, turns the catalog into a runtime dependency and puts provenance inside the data, because, as he writes, "Provenance kept in a separate log beside the data drifts from it." The retailer's customer-service agent would need order status that's minutes old and a record of which table every answer came from. If agents are on your roadmap, build for them now, since adding lineage after launch is the expensive way to learn it.

How do you keep a modernized data pipeline compliant with HIPAA, SOC 2, PCI DSS, and GDPR?

By making every control a property of the pipeline. Encryption, masking, fine-grained access, lineage and audit logs get created as data moves, so the evidence exists before an auditor asks. Bolting compliance on after cutover is the slowest and most expensive path we know.

The stakes are plain enough. IBM's 2026 Cost of a Data Breach research put the global average cost of a breach at a record $4.99 million. A migration that copies regulated data into five unmanaged S3 buckets on the way is how a modernization program ends up in that report.

The retailer's rules come from three directions. Card data from the stores and the website puts part of the estate in PCI DSS scope. California shoppers can ask it to find and delete their data under the CCPA. And the Dutch fulfillment center handles EU customers' personal data, which brings in GDPR. The figure maps common frameworks to the controls each one leans on, and the table after it shows what auditors expect to see.

Which controls does each regulation expect from your pipeline?

 

For US buyers, the hard part is usually the patchwork, since HIPAA, state privacy laws and sector rules can all touch one dataset. For European buyers, residency and erasure carry the design. Say a customer in Rotterdam asks the retailer to delete her account. GDPR gives it one month to respond, and an Iceberg row-level delete only finishes the job once the old snapshots that still hold her rows expire. On tables with personal data, we'd set snapshot expiry well inside that month and agree it with the DPO.

In financial services, we design so that, in the words of our financial services practice, "Lineage became audit-grade as a property of the system." For one regulated financial services client, that ended the contest between compliance and analytics for separate budgets. New analytics on its platform inherit the foundation's controls, so a new dashboard or model doesn't need its own compliance review.

Healthcare and life sciences teams carry extra weight: PHI, HITRUST mappings, and FDA 21 CFR Part 11 for validated systems. We treat PHI handling as a design constraint from the first sprint, which is far cheaper than reworking jobs after a security review flags them. The same logic applies to manufacturers with OT and SCADA data, where the constraint is operational continuity and a pipeline outage can stop a line.

What does data pipeline modernization on AWS cost?

It depends on four things: how many pipelines and sources you run, how much undocumented logic sits inside them, your compliance scope, and the commercial model your delivery partner signs. Until someone has measured your inventory, any quote is an estimate.

What drives the price of a pipeline modernization program?

 

At the retailer, the cheapest line on the plan is the jobs nobody reads. About 130 of its 600 jobs hadn't had a downstream read in 90 days, so they'd be switched off after owner sign-off instead of migrated.

After cutover, AWS's pricing models set the run cost. AWS Glue bills per DPU-hour, Amazon Athena and Redshift Spectrum bill by data scanned, Amazon Redshift bills by node or by serverless capacity used, and streams bill by throughput. File format, partitioning and job sizing decide which side of the budget you land on.

How should you model total cost of ownership?

Compare the full three-year picture, because a cheap migration followed by an expensive run rate isn't a saving. The model we use with finance teams adds run, change and risk for each of the three years, then adds the one-time migration cost.

The legacy column usually looks cheap on run and expensive on change and risk. At the retailer, adding a new marketplace feed last year took seven weeks of work in the old suite, and that's the kind of number to bring to the CFO.

How do companies fund the program?

Mostly from money already in the IT budget. The ETL license coming up for renewal, the warehouse support contract and the hours your team spends on break-fix are all candidates for reallocation. The AWS Migration Acceleration Program (MAP) can also offset part of the migration cost for eligible workloads, and your AWS account team can confirm what applies to you.

How should you sequence a data pipeline migration to AWS?

In five gated phases: scope and commit, design and prove, build in vertical slices, validate in parallel, then cut over and retire legacy. Each phase ends with your sign-off, so nobody discovers in month five that the plan never worked.

Legacy retirement sits inside the final gate, which keeps the old license and hosting bill from outliving the program. For the retailer, that gate has to close before the June 2027 renewal.

Which migration pattern fits each pipeline?

You won't treat every job the same way, and you shouldn't pay to. Sorting the inventory into four buckets early is the best way we know to protect the timeline. At the retailer, about 130 jobs would be retired, 70 rehosted for a short remaining life, 250 replatformed onto AWS Glue, and 150 refactored, including everything that touches revenue, inventory, card data or the AI roadmap.

How do you de-risk the cutover?

Three habits separate programs that ship from programs that stall at handoff. The first is proving the riskiest assumption early. At the retailer, that's the 3,400-line stored procedure that calculates store margin, plus the order stream the fraud team depends on, and both get built in the first few weeks. How we work is built around the same idea: "If a plan is going to break, it should break in week three on a proof rather than in month five on a deadline."

The second is running new and legacy side by side on real data. A validation harness compares row counts, aggregates and business totals between the two systems, and every variance gets an explanation before anyone signs off. For a retailer, that parallel run should cover at least one month-end close.

The third is rehearsing the cutover. A written runbook with agreed rollback criteria turns go-live weekend into a checklist. Decommissioning the old suite belongs in the plan as a deliverable, or it quietly becomes a permanent second bill.

One more point for regulated teams: keep your auditors and your DPO informed at each gate. Bringing them in at the end is how a finished pipeline sits idle for a quarter waiting on review.

Should you modernize in-house, hire a traditional consultancy, or bring in an agent-native firm?

Build in-house when you have spare senior data engineers and a forgiving deadline. Bring in outside help when the timeline is fixed, the compliance bar is high, or your team is already stretched keeping current pipelines alive. The real decision is which delivery model carries the risk of a slip.

If you've lived through a program that ran two or three times over budget, you already know the root cause usually sits in the commercial model and the delivery economics. Strong engineers inside a model that rewards billable hours will still miss the date. Our own terms are in our delivery commitment: one date, a fee fixed once scope is confirmed, and overage we absorb when a delay is ours.