Apache Iceberg on AWS: Costs, Compliance and Migration

When data leaders bring up Apache Iceberg on AWS with us, it's usually already on their roadmap, and what they want to talk about is how to get there without a two-year program, a surprise invoice, or a finding in the next audit. We've been building data platforms on AWS since 2008, a lot of them for banks, health systems, and manufacturers that answer to regulators. The point we end up making in nearly every one of those conversations is that the table format you pick now decides what your AI agents can read, what your auditors can check, and how much leverage you'll have with engine vendors for the next decade.

It matters more now that agents are reaching production. In McKinsey's August 2026 State of AI report, the share of $1 billion-plus companies scaling agents in at least one function went from 27 percent to 40 percent in a single year. We'd be wary of any of those agents working from stale, unversioned, or locked-in data, because that's where the risk shows up long before the value does.

As a running example, we'll use a typical payments company with a US business and an EU subsidiary in Dublin. About 1,800 Hive tables of transaction and merchant history sit on an on-premises Hadoop cluster whose support contract ends next year, and finance reports run off a separate warehouse. EU cardholders send erasure requests every week, and the Irish entity is in scope for DORA. Now the disputes team wants an AI agent that drafts chargeback responses from that data. Most of the choices below look different once there's a real estate like that attached.

We wrote this for the executives who'll sign off on the decision, so it sticks to what they get asked about: the architecture, whether to use S3 Tables, compliance in the US and EU, what it really costs, and the migration sequence we follow when the production date is written into the contract.

What is Apache Iceberg, and how does it work on AWS?

Apache Iceberg is an open table format. Think of it as a layer of metadata over the Parquet files in Amazon S3 that lets them act like database tables, with ACID transactions, schema changes that don't break anyone's queries, hidden partitioning, and time travel to earlier versions. Because that metadata is open, Athena, EMR, Glue, Redshift, and engines outside AWS can all read the same tables, and on AWS they find them through the AWS Glue Data Catalog. Most enterprises we work with get started fastest on Amazon S3 Tables, while teams that want to run maintenance themselves keep their tables self-managed.

What you end up with is a pile of Parquet files that behaves like governed warehouse tables without being tied to any one engine. Keeping storage separate from compute is the commercial point of the exercise, and it's why Iceberg now sits at the center of most data platform modernization programs we scope on AWS, along with the wider data modernization work around them.

Why is Apache Iceberg on AWS a board-level decision?

Our view, after a lot of platform programs, is that the table format you standardize on will outlast the warehouse, the BI tool, and quite possibly the CIO who chose them. Engines get swapped every few years. The format your petabytes sit in tends to stay, so whoever controls it controls your switching costs.

Three things have moved the question out of the architecture review and into the boardroom.

• Agents have to be able to show their work. If the disputes agent sends a chargeback response, someone may ask months later exactly what data it used, and Iceberg snapshots keep that record by design.
• Engine pricing keeps moving. With the storage open, we can route a workload to Athena, EMR, Redshift, or an outside engine on price and performance, without migrating the data each time.
• Regulators are asking about exit plans. Under DORA, financial entities need exit strategies for ICT services that support critical or important functions, and the EU Data Act's cloud switching rules have applied since September 12, 2025, with switching charges due to end on January 12, 2027. An open format turns an exit plan from a slide into something you can show.

Our Data Decides point of view series puts it plainly: “Agents read data, ground against data, and act on data. If the data they read is stale, the agent acts on stale information.” That's why we treat the table format as an AI decision first.

What does a production Iceberg lakehouse on AWS actually look like?

It has five layers, and the one most teams underinvest in is the catalog. Once the catalog and access model are right, every engine downstream inherits the same permissions and reads the same version of the data. Our guide to AWS data lake architecture goes deeper on the storage and catalog choices.

Two platform changes matter for buyers this year. Amazon S3 Tables now supports Iceberg V3, starting with deletion vectors and row lineage in November 2025, adding the variant type for semi-structured data in July 2026, and covering all V3 data types by the end of September. And since April 2026, Amazon Redshift can run UPDATE, DELETE, and MERGE straight against Iceberg tables, so the warehouse and the lake can share one copy of the data, which changes the math on any data warehouse modernization plan.

V3 isn't something we'd switch on everywhere yet. The upgrade is one-way, the variant type launched in 15 Regions, and Amazon Athena can't read V3 tables at the moment. So at the payments company, we'd leave every table the analysts query through Athena on V2 until that changes.

Should you choose Amazon S3 Tables or self-managed Iceberg tables?

We'd make S3 Tables the default for most mid-market and enterprise teams, mainly because AWS then runs compaction, snapshot management, and unreferenced file cleanup and your engineers don't have to. Self-managed Iceberg on general purpose buckets still makes sense when you need your own compaction schedules, cross-account layouts S3 Tables doesn't fit, or an existing estate you'd rather convert in place. The gap has narrowed, too, since self-managed tables registered in the AWS Glue Data Catalog can use Glue's table optimizers for compaction, snapshot retention, and orphan file deletion.

Our rule of thumb is that if nobody on your team wants to own compaction at 2 a.m., you start with S3 Tables and keep self-managed tables for the handful of datasets that justify the overhead. At the payments company, the Hive tables being converted in place would stay self-managed, and new tables would go into S3 Tables.

How does Apache Iceberg compare with Delta Lake, Apache Hudi, and a legacy warehouse?

Most of the Iceberg vs. Delta Lake evaluations we sit in on AWS end up going Iceberg's way, mainly because more AWS services read and write it natively than any other open table format. Delta Lake and Apache Hudi are both capable, and each has real strengths. For an AWS-first company, though, native service support and neutral governance of the spec tend to settle it.

If you already run a large Delta or Hudi estate, nobody is asking you to rip it out next quarter. The more useful question is where your next five years of new data will land, and on AWS we keep landing on Iceberg.

Is Apache Iceberg on AWS compliant with HIPAA, SOX, GDPR, and DORA?

Not on its own, and no table format is. Auditors look at the whole system around the data, so what matters is that Iceberg makes compliance a lot easier to prove. It keeps immutable snapshots and a full commit history, it supports row-level deletes, and it puts every engine behind one catalog.

What are the two compliance traps teams fall into with Iceberg?

The first is the erasure trap, and the payments company would hit it with its very first EU erasure request. Deleting a cardholder's rows writes a new snapshot, but the older snapshots still hold those rows until they expire and the orphan files are removed. With 90-day snapshot retention, the real erasure window is 90 days, which is how a European regulator would read it too. On S3 Tables the default is much shorter, with snapshots kept for 120 hours, but unreferenced files are only flagged after three days and deleted ten days after that, so we'd document the whole window, not just the snapshot setting.

The second is the retention trap. Teams under SEC Rule 17a-4 or similar books-and-records rules sometimes switch on S3 Object Lock under live Iceberg tables. Object Lock doesn't stop compaction, but every file cleanup after that only adds a delete marker, so storage keeps growing until the retention period ends, and S3 Tables doesn't support Object Lock at all. We'd keep a separate, locked archive for regulated records and let the working tables stay maintainable.

In regulated finance, we hold ourselves to a standard examiners recognize. As our financial services data modernization practice puts it, “Regulator-defensible means something checkable: at any point after the fact, a named person can reconstruct what happened to a given number and why.” Iceberg snapshots, paired with Lake Formation audit logs, give that person the evidence without a forensic project.

Healthcare adds its own weight. PHI handling, HITRUST mapping, and minimum-necessary access all have to hold while analysts and agents query the same tables. Lake Formation row and column filters carry the policy, and query logs tied to snapshot IDs show who saw which version of the data, which is the backbone of every healthcare data platform we put into production.

How much does Apache Iceberg on AWS cost, and where does the money leak?

The format is free, so the cost of Apache Iceberg on AWS comes down to storage, maintenance, compute, and the people who run it. In our experience, small files, snapshots nobody expires, and duplicate copies kept “just in case” sink lakehouse budgets far more often than the run-rate bill does.

AWS cut S3 Tables compaction prices in July 2025, with per-GB binpack processing 90% lower, and AWS Glue 6.0 arrived in August 2026 with a 30% price cut and full Iceberg V3 support. Those help, but they don't fix bad habits. Flexera's 2026 State of the Cloud Report found that wasted cloud spend rose to 29 percent, reversing five years of decline, with AI and newer platform services adding to the complexity. A lakehouse without maintenance discipline adds to that number every month.

Who should pay for the move?

Most of the budget you need is already inside your IT spend. The legacy warehouse license that's up for renewal, the Hadoop cluster nobody puts new workloads on, and the extract jobs that bill hours just to keep themselves alive are all candidates. For the payments company, the Hadoop support contract that ends next year is the obvious source, and if that cluster sits in a data center you're leaving anyway, it belongs in the wider data center modernization plan. Once a CFO sees the migration funded by retiring those lines, it stops being a budget increase and becomes a capital allocation call.

The commercial model matters as much as the architecture. We commit to a named ship date and a fixed fee in the statement of work and absorb the overage on delays within our control, as set out in our delivery commitment. Delays caused on the customer's side convert to time and materials at the rate disclosed upfront in the same document, so each side carries the risk it can actually manage.

How do you migrate to Apache Iceberg on AWS without stopping the business?

We'd do it in vertical slices, checking each one against the legacy system on live data before anyone is asked to switch. The programs we see run two or three times over budget are usually big-bang cutovers where validation was left until the end.

Which Apache Iceberg migration method fits your estate?

For the payments company, we'd expect most of the 1,800 Hive tables to go through migrate, the messy raw zone through add_files, and the finance warehouse through CDC until cutover.

What does the delivery sequence look like?

This is the sequence we run on every data platform engagement, with a customer signature at each exit and a forward-deployed engineer owning the judgment calls.

The full method, including how the phase gates work, is on our how we work page.

How does Apache Iceberg on AWS make your data ready for AI agents?

The short version is that every model and agent reads the same governed, versioned copy of the data, and there's enough lineage left behind to explain an answer later. The payments company's disputes agent might run on Amazon Bedrock or in its own containers. Either way, it would query the Iceberg tables through Athena or Redshift and be bound by the same Lake Formation policies as the analysts, which is the pattern we use for the production AI agents we build on AWS.

Four Iceberg features carry most of the load for agents: snapshot pinning, row lineage, the variant type, and branching. Here's how they'd play out for the disputes agent. Say a merchant contests a chargeback response in March. The agent's query was pinned to a snapshot back in January, so the team can rerun it today and get the same rows it got then, however many times the transaction tables have changed in between. With row lineage, which came with V3, every row carries an ID and the number of the commit that last touched it, and the nightly job that refreshes the agent's retrieval index uses that to re-embed only the merchants whose records changed, which is one reason we'd have the agent read its V3 tables through Redshift rather than Athena. The raw responses from the card processor can go into a variant column in the same table, so there's no separate flattening job for someone to babysit. And when the data science team wants to try a new feature for the agent, they can create a branch through Spark on EMR or Glue and test against that, leaving production alone. Our view on agentic AI in production covers the rest of what that takes.

Is your organization ready to commit? A quick scorecard

Our rough test is three or more. If at least three of the signals below sound like your team, we'd expect the move to Apache Iceberg on AWS to pay for itself within the first renewal cycle it replaces.

What should you ask any Apache Iceberg implementation partner?

If we were on the buying side, these are the six things we'd ask any implementation partner, ourselves included, before signing anything:

1. Which catalog will be the system of record, and how will you make sure Athena, Redshift, and EMR all respect its permissions?
2. Our privacy policy promises deletion within a set number of days. How does your snapshot retention plan keep us inside that, and inside our records retention rules?
3. Will you validate on live data, in parallel with the old system, or only on a test copy?
4. Once we're live, who runs compaction and cleanup, and what will that cost us each month?
5. Is the ship date in the statement of work, and who pays if it slips?
6. Is switching off the old platform in scope, or is that another project later?