Data Lakehouse Prerequisites

Before you can create or query a Data Lakehouse table from the Data Lakehouse page in C3 AI Studio or from a notebook, your C3 Application must have a Spark cluster, a catalog, and the right access for each role. This topic lists the prerequisites for each role and the supported storage backends.

Copy link to this section

Roles

The Data Lakehouse uses the following roles:

• C3.DataLakehouseUser: Reads Lakehouse tables, submits queries from the SQL Editor or a notebook, and views Spark Executions for the queries the user runs.
• C3.DisclosureRequestor: Requests access to a table snapshot from another application, including download requests.
• C3.DisclosureReviewer: Approves or rejects access requests from other applications.

A user without at least the C3.DataLakehouseUser role cannot see Lakehouse tables on the Data Lakehouse page in C3 AI Studio or query them from a notebook.

Cluster lifecycle commands (ensureService, stop, terminate) require a platform operator role. See Data Exploration with Apache Spark on C3 Agentic AI Platform for the cluster admin model.

Copy link to this section

Spark cluster

The Data Lakehouse needs an active Spark cluster for any read or write that goes through Spark. The first time you submit a query, the cluster moves from Pending or Hibernated to Running. Cold start can take several minutes.

A default cluster ships with most C3 Applications. To check the status of the default cluster:

c3.SparkCluster.inst().status

For a named cluster, use forName:

c3.SparkCluster.forName("dedicated_spark").status

If your team needs a dedicated cluster for production workloads or to isolate batch and interactive use, see Set Up a Spark Cluster.

Copy link to this section

Catalog

A catalog holds Lakehouse tables. Without a catalog, you cannot create a table.

Most C3 Applications ship with a default Hadoop catalog. To check the default:

c3.DataLake.Catalog.inst()

To list every catalog in the application:

c3.DataLake.Catalog.fetch().objs

If the default catalog does not match your storage layer or governance needs, register a new one. The Data Lakehouse supports the following catalog kinds in 8.11:

• Hadoop: File-system catalog over an object store (S3, GCS, Azure Data Lake Storage). The default for most applications.
• JDBC: Tracks Iceberg metadata in a relational database. Use this for production tenants that need a metastore.
• Snowflake: Reads Iceberg tables managed by Snowflake.
• External Application: Forwards listing, metadata reads, and table creation to the producer application's catalog. deleteTable and transferTableOwnership are not supported through this kind.

For setup steps, see Register a Data Lakehouse Catalog.

Copy link to this section

Storage backend

Hadoop catalogs need an object store. The Data Lakehouse supports the following bucket URL schemes:

Your cluster service account or workload identity must have read and write access to the bucket. Configure the access keys through the corresponding source-system config (for example, AwsS3SourceSystem, GcsSourceSystem, AzureStorageSourceSystem).

Copy link to this section

Network and runtime

For Spark Connect from an external IDE, your workstation needs:

• Outbound HTTPS access to the cluster's Spark Connect URL.
• A C3 user session token, retrieved with c3.userSessionToken().signedToken.
• The seeded py-spark_312 Jupyter runtime. For workflows outside Jupyter, install pyspark[connect]==4.1.3 locally.

For the in-platform Data.SparkSession, you do not need to install anything locally. The session runs against the Spark cluster directly.

Copy link to this section

Browser support

The C3 AI Studio Data Lakehouse page and SQL Editor support the same browsers as the rest of C3 AI Studio. Use a current version of Chrome, Edge, Firefox, or Safari.

Copy link to this section

See also

• Data Lakehouse Overview
• Data Lakehouse Quickstart
• Register a Data Lakehouse Catalog
• Data Exploration with Apache Spark on C3 Agentic AI Platform