C3 AI Documentation Home

Unstructured Data Pipeline Architecture

An Unstructured Data Integration (UDI) pipeline is a chain of connected nodes on the Data Fusion canvas. To open the canvas, select Data > Data Fusion in C3 AI Studio. Each node handles one stage, from reading documents to storing searchable passages and extracted records. Knowing the nodes and the order the canvas adds them helps you plan which nodes to add and where the output goes.

Nodes in a UDI pipeline

With the default Mew3 processing engine on the source collection, every UDI pipeline has these nodes:

  • File source system: The connection to the storage that holds your documents, such as the built-in Canonical source system or an Amazon S3 bucket. An external source system also holds the credentials to reach the storage. One source system can hold many source collections.
  • Source collection: The set of files the pipeline processes, which is every file under one folder of the source system, nested folders included.
  • Chunker: Parses each document and splits the content into chunks, the pieces of a document small enough to search on their own. The Chunker calls a large language model (LLM) through a model client to extract content.
  • Embedder: Computes an embedding for each chunk, a list of numbers that represents its meaning. The Embedder writes the embeddings to a vector store, which holds each chunk with its embedding. The Retriever ID names that vector store.
  • Genai.Vector.SourcePassage: The passage node, which stands for the passages in the vector store. A passage is a stored chunk with its embedding. The passage node's menu offers View Data.

Two nodes are optional, and a third comes with one of them:

  • Metadata Tagger: Uses an LLM to tag each document with values from categories you define. The tagger reads text sampled from the document and applies the tags to the whole document.
  • Entity Extractor: Uses an LLM and a schema you create or import to extract structured entities from the document text. Examples of entities are a contract, a person, or a facility.
  • Load: Added when you save an Entity Extractor. The Load node sends the extracted entities to your application data model or to files in external storage.

A pipeline of source collection, Chunker, Embedder, and Genai.Vector.SourcePassage indexes documents end to end. Add a Metadata Tagger when you want each document labeled with properties such as type or topic. Add an Entity Extractor when you want structured records alongside the searchable passages.

With the Doc AI processing engine, the collection has a single Strategy node instead, plus one entity node for each entity you define in it.

Order in which the canvas adds nodes

The canvas adds nodes as you configure each one. A placeholder is an unconfigured node that you select to configure that step, and saving a node creates the placeholders for the nodes that follow it:

  1. After you create a source collection, the canvas shows the collection node and a Chunker placeholder.
  2. After you save the Chunker, the canvas adds three placeholders at the same time: Entity Extractor, Metadata Tagger, and Embedder.
  3. After you save the Embedder, the canvas adds the passage node, Genai.Vector.SourcePassage.
  4. After you save the Entity Extractor, the canvas adds a Load node with one entity node for each entity you defined. An entity node stands for the stored records of one entity.

When you configure a Metadata Tagger, the canvas connects it between the Chunker and the Embedder. The Chunker, the Metadata Tagger, and the Embedder then run in series. The Entity Extractor and its Load node form a separate branch from the Chunker.

Load destinations for extracted entities

The Load node sends extracted entities to one of two destinations. After you save the Entity Extractor, the Load node uses Load to External Source until you change it:

  • Load to Application Data Model: Writes the extracted entities as records of an entity Type in your application. If the target entity Type does not exist, the platform creates it when you save the Load node. Choose this destination when your application reads the entities as records.
  • Load to External Source: Writes the extracted entities as files to the application's file storage, under the Prefix address and a Root URL folder you can change. Choose this destination when another system consumes the entities as files.

The menu on each entity node depends on the destination. With Load to External Source, the menu offers Browse Files. With Load to Application Data Model, the menu offers Edit properties, View Data, and Delete. When you edit a field on an entity node, the Entity Extractor shows the same change.

To delete an entity node in Application Data Model mode, you type the entity name in a confirmation dialog. Deleting the entity node has these effects:

  • The platform removes the entity Type from your application.
  • The canvas removes the entity node.
  • If the Load node had no other entity, the canvas also removes the Load node.
  • If the Entity Extractor no longer has any entities, it returns to a placeholder.

For the destination fields, see Configure the Load Destination.

What happens when you run a pipeline

You start a run with the Execute pipeline button on the source collection node. Each run starts two kinds of jobs, which you can follow in C3 AI Studio:

  • The sync job, which lists the collection's files, on Monitoring > Jobs > Batch.
  • The chunking job, which parses and chunks the files, on Monitoring > Jobs > MapReduce.

To check a run, open the source collection node's menu (the three-dot button) and select View run status. The dialog shows how many files are at each pipeline stage and the active jobs. For details, see Monitor Pipeline Execution.

Remove a file from a collection

The Set Source File Path dialog offers Delete for the files in a collection's folder. For the file actions, see Create an Unstructured Source Collection.

Remove deleted documents from search

When a file is deleted at the source, its passages stay in the vector store, so search can still return it. A sync that checks for removed files finds the file and marks it for removal. The Sync source files menu item and the Sync Source Collection button do not run this check. Both controls set the deleted file to Syncing and leave it there, and its passages stay in search. Do not select Execute for a deleted file. The run fails the file, and a failed file is archived without its passages being removed.

To remove deleted files from search, run this sync in the C3 AI Console:

JavaScript
Genai.SourceCollection.Utils.sync({filter: "name == '<collection name>'"}, true);

The sync marks each deleted file that was indexed as Needs Removal. The second argument, true, then processes the collection: the platform removes those files' passages from the vector store and sets each file to Archived. Once a file reaches Archived this way, search no longer returns it. A deleted file that was never indexed goes straight to Archived. The same run also indexes the collection's new and changed files.

If a deleted file went to Failed before this sync, the sync archives it but leaves its passages in search. Remove the passages with the file's ID:

JavaScript
var f = Genai.SourceFile.fetch({filter: "collection.id == '<collection id>' && metadata.fileName == '<file name>'"}).objs[0];
Genai.Vector.SourcePassage.removeAll({filter: "sourcePassage.source.id == '" + f.id + "'"}, true);

A schedule that you create with Genai.SourceCollection.Utils.createOrUpdateSyncSchedule runs the same sync. The schedule removes the passages only when its shouldProcess is true.

Several pipelines in one application

Each source collection has its own pipeline with its own settings, and the canvas can hold several pipelines side by side. Each pipeline runs independently, so you can choose a different Chunker preset, embedding model, or target entity for each one.

See also

Was this page helpful?