C3 AI Documentation Home

Understand DataLake as a Source

The C3 DataLake connector enables ingestion from platform-managed data sources in Data Fusion. Unlike external connectors, it does not require user-provided authentication or expose authentication configuration during connector setup.

When to use Data Lake as a source

Use DataLake as a Source when your data already exists within the C3 platform and is organized into catalogs and tables. Because the data is platform-managed, no user-provided credentials or authentication configuration are required, and the Auth and Advanced setup steps are skipped automatically.

A typical workflow is to ingest raw data into C3 DataLake, use Spark or other processing workflows to transform and curate the data, and then use the curated tables as source collections in downstream Data Fusion pipelines and applications.

Internal Connector Model

DataLake is treated as an internal connector. No authentication fields are displayed or required during connector configuration.

Catalog-Based Access

Data is organized into catalogs. When configuring the connector, you select a catalog that defines the set of tables available for ingestion.

Connector and Source Collection Configuration

DataLake ingestion is configured in two steps:

  • Connector (Source System) — Defines the DataLake catalog
  • Source Collection — Selects tables from the catalog to use as pipeline sources

These steps are configured separately in the Data Fusion workflow.

Table-to-Collection Mapping

Each table selected from a DataLake catalog is materialized as a DataLake Source Collection in the pipeline.

  • Each table creates a separate source collection
  • Each collection acts as an independent pipeline entry point
  • Multiple tables can be selected to generate multiple source collections

Reusable Source Collections

An existing DataLake Source Collection can be selected instead of creating a new one. This allows previously configured collections to be reused without redefining the source table.

Pipeline Integration

Once a DataLake source collection is created, it can be connected to downstream nodes:

  • Transform nodes — Apply data transformation logic
  • Target nodes — Load data into destination types

Data Preview

When selecting source tables, you can preview table contents before creating the DataLake Source Collection.

Preview availability and behavior may vary depending on the current release and environment configuration.

Key Characteristics

  • No authentication configuration required
  • Catalog-driven table selection
  • Table-level pipeline entry points
  • Ability to select existing source collections

Connect to DataLake as a Source

Follow these steps to create a DataLake connector in Data Fusion.

Step 1 — Navigate to Data Fusion

  1. Open the application.
  2. From the left navigation pane, select Data Fusion under the DATA section.
  3. In the Data Fusion canvas, click + Add Data Source.

Step 2 — Select the DataLake Connector

  1. In the Configure Connector dialog, search for C3 DataLake.
  2. Select the C3 DataLake connector from the Database & data warehouse category.
  • The DataLake connector does not require authentication configuration.
  • The Auth and Advanced setup steps are skipped automatically.

Step 3 — Configure Connector Details

In the Destination step, provide the following information:

  • name — A unique connector name
  • description — (Optional) Connector description
  • catalog — Select the DataLake catalog to use for this data source

The selected catalog determines the tables available in later configuration steps.

Click Continue.

Step 4 — Review Connector Configuration

  1. Review the connector configuration in the Review & Test step.
  2. Verify:
    • a. Connector name
    • b. Selected catalog

Step 5 — Save and Test the Connector

  1. Click Save & Test to validate the connector configuration.
  2. After validation completes successfully, click Save & Close.

The DataLake connector is now available in Data Fusion and can be used to create DataLake source collections for pipeline development.

Step 6 — Configure a DataLake Source Collection

  1. In the Data Fusion canvas, click the Table node connected to the DataLake connector.

The Select Source Tables dialog opens.

  1. To create a new source collection:
    • a. Select Create new DataLake Source Collection
    • b. Enter a collection name
    • c. (Optional) Enter a description
  2. Alternatively, to reuse an existing source collection:
    • a. Select Use an existing DataLake Source Collection to reuse a previously configured collection
    • b. Choose an existing collection from the dropdown list

Step 7 — Select Source Tables

  1. Select one or more tables from the catalog.
  • Each selected table creates a separate DataLake Source Collection
  • Each collection acts as an independent pipeline entry point
  1. (Optional) Select a table to preview its contents.
  2. Click Save.

The DataLake source collection is configured and ready to be connected to downstream transform and target nodes in the Data Fusion pipeline.

Step 8 — Add Downstream Nodes

  1. Add a Transform node to apply logic.
  2. Add a Target node to define the destination.
  3. Connect nodes to complete the pipeline.

Step 9 — Preview Data (Optional)

Available only if preview functionality is enabled.

  1. Open the Transform node.
  2. Use:
  • Preview Source to view raw data
  • Preview Transform to validate transformation logic

Result

You have created a DataLake connector and configured one or more source collections that can be used to build and run pipelines.

Was this page helpful?