Data Lakehouse

What Is a Data Lakehouse?

A data lakehouse is an analytics architecture that stores tables as open-format files, usually Parquet, in object storage such as Amazon S3, Google Cloud Storage, or Azure Blob Storage, and adds a table format and catalog that provide ACID transactions, schema enforcement, and time travel. Multiple query engines read and write the same copy of the data.

Lakehouses let BI, data engineering, and machine learning share one set of files instead of copying data into a proprietary warehouse.

How a Data Lakehouse Works

A lakehouse table is a set of immutable data files plus metadata recording which files belong to the table at each snapshot.

• Data files are Parquet, ORC, or Avro files, never modified in place.
• Table metadata lists each snapshot's data files with per-file column statistics. In Iceberg, a metadata JSON file points to a manifest list, which points to manifests that list data files. Delta Lake keeps an ordered log of JSON commits in _delta_log, compacted into Parquet checkpoints.
• The catalog maps a table name to its current metadata. An Iceberg commit atomically swaps that pointer, and a Delta commit atomically writes the next numbered log file, so readers never see a partial write.

Concurrent writers use optimistic concurrency: a writer that loses the swap retries against the new snapshot. Row-level changes use one of two strategies:

• Copy-on-write rewrites every file containing a changed row: fast reads, expensive writes.
• Merge-on-read writes delete files or deletion vectors that readers apply at query time: cheap writes, slower reads until compaction.

Data Lakehouse vs Data Lake vs Data Warehouse

A lakehouse puts a transactional table layer over open files that many engines share; a data lake has no such layer, and a warehouse keeps data in a proprietary format. Historically, the data lake vs data warehouse decision meant choosing between cheap, open storage and reliable SQL.

When to use which: a warehouse fits one primary engine and SQL-centric BI, a lakehouse fits several engines on shared data, and a plain lake fits raw landing zones. For the warehouse side, see how to choose a data warehouse.

Data Lakehouse Architecture

Lakehouse architecture has five replaceable layers: object storage, a file format (usually Parquet), a table format, a catalog, and compute engines such as Spark, Trino, or Snowflake. The table format defines snapshots, schema, and the commit protocol:

• Apache Iceberg originated at Netflix and has the broadest engine support. Its v3 spec adds deletion vectors and row lineage; engine support for v3 is still uneven as of November 2026. For a deeper walkthrough, see how Iceberg catalogs, manifests, and snapshots work.
• Delta Lake was created by Databricks and is its default table format; its UniForm feature also writes Iceberg metadata so Iceberg readers can query Delta tables.
• Apache Hudi originated at Uber for upsert-heavy ingestion.
• DuckLake, released by the DuckDB team in 2025, stores table metadata in a SQL database instead of metadata files.

The catalog layer is converging on the Iceberg REST catalog API, implemented by Apache Polaris, Databricks Unity Catalog, AWS Glue Data Catalog, and Snowflake Horizon.

Data Lakehouse Use Cases

Lakehouses suit workloads where several engines share data or where storage volume dominates cost.

• Multi-engine analytics: Spark builds tables, Trino or Snowflake serves SQL, and DuckDB or Polars reads the same files from notebooks.
• High-volume append data: clickstream, logs, and telemetry.
• ML training data: a training run pins a snapshot ID and can be reproduced while that snapshot is retained.
• Database replicas: change data capture pipelines merge row changes from Postgres or MySQL into lakehouse tables.
• ELT on open storage: raw data lands first and transforms run in the engine, the ELT pattern applied to object storage.

Data Lakehouse Challenges

The main cost of a lakehouse is table maintenance that a warehouse performs automatically.

• Small files: frequent commits bloat file counts and metadata, slowing queries until compaction runs.
• Storage growth: old snapshots keep their files, including deleted rows, until snapshot expiration runs.
• CDC read cost: in merge-on-read tables, upserts create delete files on every commit. Flink's Iceberg upsert mode writes equality deletes, which readers match against every older data file they scan until compaction.
• Commit contention: many concurrent writers on one table cause retries.
• Interactive performance: warehouses still win on high-concurrency dashboards and selective queries, because they cache results and keep metadata in their own service instead of reading it from object storage.