C3 AI Documentation Home

Data Integration Overview

Your C3 application needs data that lives in other systems, such as files in cloud storage, database tables, event streams, and documents. Data integration brings that data into the application and keeps it current, so the application always has records it can query. The C3 Agentic AI Platform supports two kinds of data integration: structured, for files, tables, and event streams, and unstructured, for documents.

To bring data in, you build a pipeline for each source. Before you build a pipeline, you choose structured or unstructured integration. You also choose between code and Data Fusion, the visual pipeline editor in C3 AI Studio. One pipeline, from a CSV file to stored records, shows the parts of a typical structured pipeline.

Why data integration uses pipelines

Most applications draw data from many systems: enterprise resource planning (ERP) systems, data warehouses, files on cloud storage, event streams, document repositories, and APIs. Each system names and shapes its data differently. When an application reads each source directly, every change in a source breaks the application code that reads it.

The platform routes every source through a pipeline instead. A pipeline is the path the platform runs to read data from a source and reshape it for your application. Most pipelines store the result as records in the application; a virtualized pipeline queries the source table in place. When a source changes, you update the pipeline, and the code that reads the records keeps working.

Example: a CSV file becomes records

In this example, the order history for your application arrives as CSV files in an Amazon S3 bucket. A structured pipeline for those files has five parts, which you create in this order:

  1. A source system records where the data lives, here the S3 bucket, and holds the credentials to reach it.
  2. A source collection is one dataset in that source system, here the folder of order files.
  3. A Source Type describes the columns of one incoming row exactly as they arrive, such as an order ID, an amount, and an order date.
  4. A transform maps each incoming row to the fields of your target entity Type, for example an Order Type. An entity Type is a C3 AI Type whose records the platform stores.
  5. The target entity Type holds the stored records. Your application reads the stored records with fetch() and forId().

Each run of the pipeline reads the files, applies the transform, and stores the resulting records. The platform records the outcome of every run in status records, so you can check what loaded and what failed.

Structured pipelines support four integration techniques:

  • Ingest: Copies the data into your application in bulk runs.
  • Incremental ingest: Copies only the rows that changed since the last run.
  • Streaming: Copies each message from an event stream as it arrives.
  • Virtualization: Queries a database table in place, without copying it.

Ingest, incremental ingest, and streaming pipelines share these five parts and record every run in the same status records, whatever the source. A virtualized pipeline uses fewer parts: a source system, a source collection that names the table, and an External entity Type that matches the table. An External entity Type is a Type whose records stay in the external database. A virtualized pipeline has no transform and stores no records.

Data Integration End-to-End Workflow describes each component and technique in detail.

Choose structured or unstructured data integration

Choose by the shape of your source data. Data with a fixed set of columns needs structured integration, and documents need unstructured integration.

Structured Data Integration (SDI) loads data that has a fixed schema: database tables, CSV, JSON, Parquet, Avro, ORC, or Excel .xlsx files, and event-stream topics. SDI maps columns to the fields of your entity Types with transforms. SDI can infer a Source Type from a sample of your data, a step called type inference. SDI also offers change data capture (CDC), the mechanism behind incremental ingest for database tables. You read the results from your entity Types with fetch() and forId().

Unstructured Data Integration (UDI) loads documents such as PDF, Word, PowerPoint, spreadsheet, HTML, Markdown, and image files. UDI parses each document and splits it into chunks, short passages that a search can match. UDI can also attach metadata and named items that a large language model (LLM) extracts from each chunk. UDI stores each chunk as an embedding, a numeric form of its meaning, which powers semantic search and retrieval-augmented generation (RAG).

A spreadsheet can take either route. To load an Excel workbook's rows into entity Types, read the workbook through a file source with SDI, as described in Configure a Source Collection. To search the workbook as a document, load it with UDI.

If your data includes both tables and documents, you need both kinds of pipeline. Build a structured pipeline for each table or file source, and an unstructured pipeline for the documents. Both kinds of pipeline can use the same source system to reach a storage location.

Build on the Data Fusion canvas or in code

You can build a pipeline on a visual canvas or in code. Both routes create the same platform objects, so you can start on the canvas and later inspect or change the pipeline from code:

  • Data Fusion: The visual pipeline editor in C3 AI Studio. Open your application in C3 AI Studio, then select Data > Data Fusion. Each part of a pipeline appears on the canvas as a node. You add the first node with Add Data Source. After you configure a node, the canvas attaches a placeholder for the next part; select the placeholder to configure that part. Structured and unstructured pipelines both use this canvas.
  • Programmatic: Commands in the C3 AI Console and package files in your IDE. Package files are your application's package source, which the platform syncs to the application. You declare the source system, source collection, Source Types, and transform files in your package. You review the package files like any other code and promote the same version across environments. In code you can also run BatchJobs, platform jobs that process data in batches, before and after a pipeline.

To build on the canvas, start with Connect with File Source System. For the programmatic route, see Integrate Data Programmatically. For BatchJobs that run around a pipeline, see Run BatchJobs Before and After a Pipeline.

Where to start

Pick the entry that matches your goal:

Was this page helpful?