C3 AI Documentation Home

Prerequisites for Unstructured Data Integration

Before you build an Unstructured Data Integration (UDI) pipeline, confirm the following.

Application access

  • You have access to a C3 Agentic AI Platform application with Data Fusion enabled.

  • You can open the Data > Data Fusion page in C3 AI Studio.

  • Developer access to the target application, which provides access to the Data Fusion workspace and allows configuration of Source Systems and Source Collections.

  • Task nodes used for UDI processing should have at least:

    • 30 GB memory
    • 5 CPUs
    • 100 GB disk space

    Insufficient resources may cause processing failures or node crashes during document processing workloads.

Roles and permissions

Assign one of the following role combinations based on what each user must do:

  • Pipeline builder: C3.AppAdmin (or C3.EnvAdmin) and C3.StudioUser. Creates, edits, and runs UDI pipelines, source systems, and source collections through the Studio UI. Add C3.ConsoleUser if any setup steps run from the Console.
  • Result consumer: C3.StudioUser and the read permissions granted by the application's standard user role. Views existing pipelines and searches or chats over the indexed content.

You assign roles in Administrative > User Management.

A source for your documents

UDI consumes files from a FileSourceSystem. You have two options:

  • The platform's built-in Canonical source system. No external cloud credentials are required. Use it for quick tests, small document sets, and uploads through the UI.
  • An external file source system: Amazon S3, Azure Blob Storage, Google Cloud Storage, OneDrive or SharePoint, or Google Drive. Use one for production document libraries that already live in cloud storage.

A model client

UDI uses LLM clients for parsing in the Chunker and, optionally, for metadata extraction and entity extraction. Confirm that at least one model client is configured in Model Hub > Model Clients. The default default-completions client works for the Quickstart.

Vector store and Retriever ID

Each UDI pipeline writes embeddings to a vector store identified by a Retriever ID. The default default-pg Retriever ID is created automatically. If you plan to use the embeddings in GenAI Search or another retrieval workflow, plan a Retriever ID that downstream services reference.

See also

Was this page helpful?