C3 AI Documentation Home

C3 AI Data Lakehouse Table Branching

The C3 AI Data Lakehouse APIs are backed by Apache Iceberg and support table-level branching. A branch is a named line of development for the same logical table. You pass an optional branch argument on read and write APIs; the default branch is "main". Branching lets teams experiment with data and schema changes in isolation from main.

Branching runs from a notebook — the C3 AI Studio Data Lakehouse page does not expose branch creation, branch reads, or branch removal. The code samples below use the Python API.

The C3 AI Data Lakehouse Python API in 8.11 does not provide branch merging or branch deletion. Branches are intended for isolated reads and staged writes. To retire a branch, drop the underlying table with table.remove() (which removes all branches) and recreate it with the data you want to keep. Iceberg SQL procedures such as fast_forward and ALTER TABLE … DROP BRANCH are not C3-supported against federated Lakehouse tables.

Key concepts

ConceptDescription
Branch parameterAll read and write APIs accept an optional branch string. The default is "main".
Auto-creationWrite operations automatically create the branch if it does not exist.
Data isolationData written to a branch is isolated from main.
Schema isolationchangeSchema on a non-main branch updates the schema for that branch only.

Prerequisites

  • A C3 AI Data Lakehouse-enabled application and DataLake.Catalog access.
  • Familiarity with Apache Iceberg branches and snapshots.
  • For CSV import examples, files reachable from the cluster (for example uploaded to server-accessible storage).

End-to-end pattern

A typical exercise:

  1. Configure catalog, namespace, table name, and a feature branch label.
  2. Create an Iceberg table (or fetch it if it already exists).
  3. Import CSV to main, then to feature with different data — the branch is created on first write if needed.
  4. Read each branch and compare row counts, keys, and values.
  5. Fetch metadata per branch (currentSnapshot, dataSchema, optional snapshots) to see divergence.
  6. Optionally changeSchema on the feature branch, append data with the new columns on feature, and append to main using only the original columns.

Runnable code for all steps is available in the tutorial notebook Data Lake Branching Overview.

Write to a branch

Before importing, create a table with a schema that matches your CSV columns. Then pass cluster-readable URLs (for example gcs://, s3://) to importCsv. Use the same pattern in the C3 Data Lake Snapshot Versioning and Management and Share Data Lakehouse Tables Across Applications examples. Omit the branch parameter to write to main, or pass a branch name (for example feature) to write to another branch.

Python
csv_spec = c3.CsvParsingSpec.builder().header(True).build()

# Write to feature branch
branch_csv_url = "gcs://c3--datasets/WindTurbine_1000assets_3GB/WindTurbine/CanonicalWindTurbineMeasurement/inbox/measurements-10.csv"
table.importCsv(branch_csv_url, c3.DataLake.Table.ImportMode.OVERWRITE, csv_spec, "feature")

# Write to main branch (default - branch parameter omitted)
main_csv_url = "gcs://c3--datasets/WindTurbine_1000assets_3GB/WindTurbine/CanonicalWindTurbineMeasurement/inbox/measurements-1.csv"
table.importCsv(main_csv_url, c3.DataLake.Table.ImportMode.OVERWRITE, csv_spec)

Read from a branch

Pass branch as the last argument to readArrowIterator after asOf. Use None for asOf when you are not time-traveling.

Python
main_df = table.readArrowIterator("", [], 1000, None, "main").read_pandas()
feature_df = table.readArrowIterator("", [], 1000, None, "feature").read_pandas()

Other read APIs that accept branch include readAsTuples and readPartitions.

Compare branches

Use DataLake.Catalog.TableSpec with .branch(...) and optionally .includeSnapshots(True) when calling catalog.table. Inspect currentSnapshot, dataSchema, and snapshots to compare branches after independent writes.

Python
# Fetch feature branch metadata

feature_spec = c3.DataLake.Catalog.TableSpec.builder() \
    .namespace(namespace) \
    .branch("feature") \
    .includeSnapshots(True) \
    .build()

feature_table = catalog.table(table_name, feature_spec)

# Fetch main branch metadata
main_spec = c3.DataLake.Catalog.TableSpec.builder() \
      .namespace(namespace) \
      .branch("main") \
      .includeSnapshots(True) \
      .build()

main_table = catalog.table(table_name, main_spec)

# Compare snapshots
print(f"Main snapshot:    {main_table.currentSnapshot.snapshotId}")
print(f"Feature snapshot: {feature_table.currentSnapshot.snapshotId}")

Schema changes on a branch

Use changeSchema(update, branch) on a non-main branch to add columns (or evolve schema) for that branch. changeSchema(update, branch) supports comprehensive schema modifications including adding columns, removing columns, renaming columns, and changing column data types. main can remain on the older schema while you append rows that include new columns only on the feature branch. main continues to accept imports that match its existing columns.

Python
# Add a column to the feature branch only
add_col = c3.DataLake.Table.SchemaUpdate.AddColumn.make() \
    .withColName("category") \
    .withColumnType(c3.PrimitiveType.ofStr()) \
    .withRequired(False)

table.changeSchema(add_col, "feature")
# main branch schema remains unchanged

# Now feature branch accepts data with the new column
# CSV with columns: id,name,value,category
table.importCsv(feature_csv_url, c3.DataLake.Table.ImportMode.APPEND, csv_spec, "feature")

# main branch still uses original schema (id,name,value only)
table.importCsv(main_csv_url, c3.DataLake.Table.ImportMode.APPEND, csv_spec)

Inspect branches with SQL

The Iceberg metadata tables expose branch information through Spark Connect:

SQL
SELECT * FROM datasets.dfl.turbineMeasurements.refs WHERE type = 'BRANCH'

The C3 AI Studio Table details modal also shows the current branch for the selected table. The 8.11 release exposes the branch as a read-only field; the full branch list is queryable through SQL.

Branch lifecycle and table removal

Removing the underlying Iceberg table removes the table and all of its branches. There is no way to recover a branch after table.remove().

Python
table.remove()

See also

Was this page helpful?