When data leaders bring up Apache Iceberg on AWS with us, it's usually already on their roadmap, and what they want to talk about is how to get there without a two-year program, a surprise invoice, or a finding in the next audit. We've been building data platforms on AWS since 2008, a lot of them for banks, health systems, and manufacturers that answer to regulators. The point we end up making in nearly every one of those conversations is that the table format you pick now decides what your AI agents can read, what your auditors can check, and how much leverage you'll have with engine vendors for the next decade.
It matters more now that agents are reaching production. In McKinsey's August 2026 State of AI report, the share of $1 billion-plus companies scaling agents in at least one function went from 27 percent to 40 percent in a single year. We'd be wary of any of those agents working from stale, unversioned, or locked-in data, because that's where the risk shows up long before the value does.
As a running example, we'll use a typical payments company with a US business and an EU subsidiary in Dublin. About 1,800 Hive tables of transaction and merchant history sit on an on-premises Hadoop cluster whose support contract ends next year, and finance reports run off a separate warehouse. EU cardholders send erasure requests every week, and the Irish entity is in scope for DORA. Now the disputes team wants an AI agent that drafts chargeback responses from that data. Most of the choices below look different once there's a real estate like that attached.
We wrote this for the executives who'll sign off on the decision, so it sticks to what they get asked about: the architecture, whether to use S3 Tables, compliance in the US and EU, what it really costs, and the migration sequence we follow when the production date is written into the contract.
Apache Iceberg is an open table format. Think of it as a layer of metadata over the Parquet files in Amazon S3 that lets them act like database tables, with ACID transactions, schema changes that don't break anyone's queries, hidden partitioning, and time travel to earlier versions. Because that metadata is open, Athena, EMR, Glue, Redshift, and engines outside AWS can all read the same tables, and on AWS they find them through the AWS Glue Data Catalog. Most enterprises we work with get started fastest on Amazon S3 Tables, while teams that want to run maintenance themselves keep their tables self-managed.
What you end up with is a pile of Parquet files that behaves like governed warehouse tables without being tied to any one engine. Keeping storage separate from compute is the commercial point of the exercise, and it's why Iceberg now sits at the center of most data platform modernization programs we scope on AWS, along with the wider data modernization work around them.
Our view, after a lot of platform programs, is that the table format you standardize on will outlast the warehouse, the BI tool, and quite possibly the CIO who chose them. Engines get swapped every few years. The format your petabytes sit in tends to stay, so whoever controls it controls your switching costs.
Three things have moved the question out of the architecture review and into the boardroom.
Our Data Decides point of view series puts it plainly: “Agents read data, ground against data, and act on data. If the data they read is stale, the agent acts on stale information.” That's why we treat the table format as an AI decision first.
It has five layers, and the one most teams underinvest in is the catalog. Once the catalog and access model are right, every engine downstream inherits the same permissions and reads the same version of the data. Our guide to AWS data lake architecture goes deeper on the storage and catalog choices.
Two platform changes matter for buyers this year. Amazon S3 Tables now supports Iceberg V3, starting with deletion vectors and row lineage in November 2025, adding the variant type for semi-structured data in July 2026, and covering all V3 data types by the end of September. And since April 2026, Amazon Redshift can run UPDATE, DELETE, and MERGE straight against Iceberg tables, so the warehouse and the lake can share one copy of the data, which changes the math on any data warehouse modernization plan.
V3 isn't something we'd switch on everywhere yet. The upgrade is one-way, the variant type launched in 15 Regions, and Amazon Athena can't read V3 tables at the moment. So at the payments company, we'd leave every table the analysts query through Athena on V2 until that changes.
We'd make S3 Tables the default for most mid-market and enterprise teams, mainly because AWS then runs compaction, snapshot management, and unreferenced file cleanup and your engineers don't have to. Self-managed Iceberg on general purpose buckets still makes sense when you need your own compaction schedules, cross-account layouts S3 Tables doesn't fit, or an existing estate you'd rather convert in place. The gap has narrowed, too, since self-managed tables registered in the AWS Glue Data Catalog can use Glue's table optimizers for compaction, snapshot retention, and orphan file deletion.
Our rule of thumb is that if nobody on your team wants to own compaction at 2 a.m., you start with S3 Tables and keep self-managed tables for the handful of datasets that justify the overhead. At the payments company, the Hive tables being converted in place would stay self-managed, and new tables would go into S3 Tables.
Most of the Iceberg vs. Delta Lake evaluations we sit in on AWS end up going Iceberg's way, mainly because more AWS services read and write it natively than any other open table format. Delta Lake and Apache Hudi are both capable, and each has real strengths. For an AWS-first company, though, native service support and neutral governance of the spec tend to settle it.
If you already run a large Delta or Hudi estate, nobody is asking you to rip it out next quarter. The more useful question is where your next five years of new data will land, and on AWS we keep landing on Iceberg.
Not on its own, and no table format is. Auditors look at the whole system around the data, so what matters is that Iceberg makes compliance a lot easier to prove. It keeps immutable snapshots and a full commit history, it supports row-level deletes, and it puts every engine behind one catalog.
The first is the erasure trap, and the payments company would hit it with its very first EU erasure request. Deleting a cardholder's rows writes a new snapshot, but the older snapshots still hold those rows until they expire and the orphan files are removed. With 90-day snapshot retention, the real erasure window is 90 days, which is how a European regulator would read it too. On S3 Tables the default is much shorter, with snapshots kept for 120 hours, but unreferenced files are only flagged after three days and deleted ten days after that, so we'd document the whole window, not just the snapshot setting.
The second is the retention trap. Teams under SEC Rule 17a-4 or similar books-and-records rules sometimes switch on S3 Object Lock under live Iceberg tables. Object Lock doesn't stop compaction, but every file cleanup after that only adds a delete marker, so storage keeps growing until the retention period ends, and S3 Tables doesn't support Object Lock at all. We'd keep a separate, locked archive for regulated records and let the working tables stay maintainable.
In regulated finance, we hold ourselves to a standard examiners recognize. As our financial services data modernization practice puts it, “Regulator-defensible means something checkable: at any point after the fact, a named person can reconstruct what happened to a given number and why.” Iceberg snapshots, paired with Lake Formation audit logs, give that person the evidence without a forensic project.
Healthcare adds its own weight. PHI handling, HITRUST mapping, and minimum-necessary access all have to hold while analysts and agents query the same tables. Lake Formation row and column filters carry the policy, and query logs tied to snapshot IDs show who saw which version of the data, which is the backbone of every healthcare data platform we put into production.
The format is free, so the cost of Apache Iceberg on AWS comes down to storage, maintenance, compute, and the people who run it. In our experience, small files, snapshots nobody expires, and duplicate copies kept “just in case” sink lakehouse budgets far more often than the run-rate bill does.
AWS cut S3 Tables compaction prices in July 2025, with per-GB binpack processing 90% lower, and AWS Glue 6.0 arrived in August 2026 with a 30% price cut and full Iceberg V3 support. Those help, but they don't fix bad habits. Flexera's 2026 State of the Cloud Report found that wasted cloud spend rose to 29 percent, reversing five years of decline, with AI and newer platform services adding to the complexity. A lakehouse without maintenance discipline adds to that number every month.
Most of the budget you need is already inside your IT spend. The legacy warehouse license that's up for renewal, the Hadoop cluster nobody puts new workloads on, and the extract jobs that bill hours just to keep themselves alive are all candidates. For the payments company, the Hadoop support contract that ends next year is the obvious source, and if that cluster sits in a data center you're leaving anyway, it belongs in the wider data center modernization plan. Once a CFO sees the migration funded by retiring those lines, it stops being a budget increase and becomes a capital allocation call.
The commercial model matters as much as the architecture. We commit to a named ship date and a fixed fee in the statement of work and absorb the overage on delays within our control, as set out in our delivery commitment. Delays caused on the customer's side convert to time and materials at the rate disclosed upfront in the same document, so each side carries the risk it can actually manage.
We'd do it in vertical slices, checking each one against the legacy system on live data before anyone is asked to switch. The programs we see run two or three times over budget are usually big-bang cutovers where validation was left until the end.
For the payments company, we'd expect most of the 1,800 Hive tables to go through migrate, the messy raw zone through add_files, and the finance warehouse through CDC until cutover.
This is the sequence we run on every data platform engagement, with a customer signature at each exit and a forward-deployed engineer owning the judgment calls.
The full method, including how the phase gates work, is on our how we work page.
The short version is that every model and agent reads the same governed, versioned copy of the data, and there's enough lineage left behind to explain an answer later. The payments company's disputes agent might run on Amazon Bedrock or in its own containers. Either way, it would query the Iceberg tables through Athena or Redshift and be bound by the same Lake Formation policies as the analysts, which is the pattern we use for the production AI agents we build on AWS.
Four Iceberg features carry most of the load for agents: snapshot pinning, row lineage, the variant type, and branching. Here's how they'd play out for the disputes agent. Say a merchant contests a chargeback response in March. The agent's query was pinned to a snapshot back in January, so the team can rerun it today and get the same rows it got then, however many times the transaction tables have changed in between. With row lineage, which came with V3, every row carries an ID and the number of the commit that last touched it, and the nightly job that refreshes the agent's retrieval index uses that to re-embed only the merchants whose records changed, which is one reason we'd have the agent read its V3 tables through Redshift rather than Athena. The raw responses from the card processor can go into a variant column in the same table, so there's no separate flattening job for someone to babysit. And when the data science team wants to try a new feature for the agent, they can create a branch through Spark on EMR or Glue and test against that, leaving production alone. Our view on agentic AI in production covers the rest of what that takes.
Our rough test is three or more. If at least three of the signals below sound like your team, we'd expect the move to Apache Iceberg on AWS to pay for itself within the first renewal cycle it replaces.
If we were on the buying side, these are the six things we'd ask any implementation partner, ourselves included, before signing anything: