Data Lakehouse Overview
The C3 AI Data Lakehouse stores large volumes of tabular data in open Apache Iceberg tables and exposes them to Apache Spark, the C3 AI Studio SQL Editor, and the C3 AI Data.SparkSession Python API. A Data Lakehouse keeps the cost profile of object storage and adds database-style features on top: schema, snapshots, branches, and ACID writes.
A Data Lakehouse is organized as catalogs, namespaces, tables, snapshots, branches, and tags.
The Data Lakehouse page in C3 AI Studio covers table creation, catalog browsing, SQL queries, Spark execution monitoring, and shared-dataset requests. Snapshot expiration, branch management, and merge-mode writes run from a notebook.
When to use the Data Lakehouse
The C3 Agentic AI Platform offers several storage options. Use the comparison below to decide which fits your data.
| Storage option | Best for | Example | Trade-off |
|---|---|---|---|
| Persistable Type (Cassandra or PostgreSQL) | Operational entity data, frequent point reads and writes, transactional updates | User accounts, asset registry | Cost scales with volume; not designed for bulk analytics |
| Data Lakehouse | Large tabular datasets, columnar analytics, time-travel queries, schema evolution without rewrites | Sensor measurements over years, billions of rows | Higher per-query latency than a database; reads need a Spark cluster |
| Vector Store | Embeddings for semantic search and retrieval-augmented generation | Document search, RAG context retrieval | Specialized to vector similarity; not for general analytics |
| External Database Connector | Read data that already lives in Snowflake | Snowflake shipment data joined with Order entities | Read-only; query latency depends on the external system |
Choose the Data Lakehouse when you need:
- Cheap storage of billions of rows of tabular data.
- Columnar analytics with Spark SQL or Pandas-on-Spark.
- Snapshot-based point-in-time queries.
- Schema changes on a branch, with no impact on production reads.
- Cross-application sharing of analytical datasets.
Capabilities
The Data Lakehouse provides the following capabilities:
- Multiple catalog types: Hadoop, JDBC, Snowflake, and an external-application catalog that grants read-only access to another C3 application's tables.
- Apache Iceberg tables: Open table format with full schema, partition expressions, primary keys, and ACID transactions.
- Snapshots and time travel: Every write produces a new snapshot. You can query any past snapshot or expire old ones to control storage cost.
- Branches: Iceberg branches let you stage writes, experiment with schema changes, or stage a release without touching the
mainbranch. - Tags: Named pointers to snapshots, useful for marking releases or known-good states.
- Spark integration: The C3 AI SparkCluster runs Apache Spark in the C3 Agentic AI Platform. You can use Spark Connect from any IDE, or use
Data.SparkSessionfor a Pandas-on-Spark API with C3 helpers. - SQL Editor: A query interface in C3 AI Studio for running SQL against Lakehouse tables, with execution history and a results preview.
- Spark Executions monitoring: Every query is tracked as a SparkExecution record, viewable from C3 AI Studio's Jobs section.
- Managed Spark clusters: A
SparkCluster.Managedinstance hibernates after an idle timeout and resumes on demand, so you pay for compute only when you query.
How the Data Lakehouse fits with Data Fusion
The Data Lakehouse is a target and a source for Data Fusion pipelines.
- A Structured Data Integration pipeline can write its output to a Lakehouse table using a Data Lakehouse load step.
- A Lakehouse table can serve as the source of a Data Fusion pipeline through DataLakeSourceCollection and DataLakeSourceSystem.
- A Source File from Data Fusion can be loaded directly into a Lakehouse table with DataLake.Operations.