C3 AI Documentation Home

Prerequisites for Unstructured Data Integration

Before you build an Unstructured Data Integration (UDI) pipeline in C3 AI Data Fusion, check that your access, environment, and document source meet these requirements.

Application access

You build UDI pipelines in Data Fusion, which is part of C3 AI Studio for your application. To check your access, open C3 AI Studio for the application and select Data > Data Fusion. The Data Fusion canvas opens on the Data Integration tab. If Data Fusion does not appear in the navigation, ask your administrator for one of the roles in Roles and permissions.

Roles and permissions

To open Data Fusion and build a UDI pipeline, you need a role that shows Data Fusion in C3 AI Studio and a UDI role. For which roles to grant, see Data Integration Roles and Permissions.

An administrator adds users to roles in C3 AI Studio from Administrative > User Management.

Task-node resources

Task nodes are the servers that run pipeline work, such as parsing and embedding documents, in the background. Each task node that processes UDI documents needs at least:

  • 30 GB of memory
  • 5 CPUs
  • 100 GB of disk space

With fewer resources, document processing can fail or a task node can stop during processing. Your platform administrator sets the task-node size, so ask them to confirm these minimums before you run a pipeline.

A source for your documents

A UDI pipeline reads files from a file source system, which is a connection to the storage that holds your documents. Choose one of these sources:

  • The built-in Canonical source system: Stores files in the application's own file storage and needs no external credentials. Choose Canonical when you upload documents yourself through the Data Fusion UI, for example to try a pipeline on a few documents.
  • An external file source system: Amazon S3, Azure Blob Storage, Google Cloud Storage, OneDrive or SharePoint, or Google Drive. Choose an external source when your documents already live in that storage. You need credentials for the source.

For both options, see Create an Unstructured Source Collection. For the credentials each external source accepts, see Authenticate File Sources.

Model clients

A model client is the connection to a hosted model that the pipeline calls. UDI uses large language model (LLM) clients to parse documents in the Chunker. If you add a Metadata Tagger or an Entity Extractor, UDI also uses LLM clients to tag metadata and extract entities. The Embedder uses an embedding model client to compute embeddings. To check which clients exist, open C3 AI Studio and select Model Hub > Clients. The Model Clients page lists every client configured for the application.

The Chunker and the Metadata Tagger use the default-completions client unless you select another one. The default embedding model depends on the clients configured in your environment.

Vector store and Retriever ID

The Embedder writes embeddings to a vector store. An embedding is a list of numbers that represents the meaning of a piece of a document. A vector store holds each passage with its embedding so that search can find it. The Retriever ID names that vector store. The default-pg Retriever ID exists in every application that includes UDI, and the Embedder's Retriever ID field is prefilled with it.

Pipelines with the same Retriever ID share one vector store, and search returns passages from all of them. Decide which ID to use before you configure the Embedder:

  • Keep default-pg when you want this pipeline's passages in the same store as other pipelines that use it.
  • Enter your own ID when you want this pipeline's passages kept separate. A search or chat feature that reads these passages then needs that same ID.

For the field, see Configure the Embedder.

See also

Was this page helpful?