C3 AI Documentation Home

The Data Integration Interface in C3 AI Studio

The Data Integration page in C3 AI Studio is a visual interface for integrating data into your application. Data Integration pulls data from sources such as databases or APIs into a single application-specific model. For more information, see Data Integration Techniques Overview.

The Data Integration page has two sections:

  1. Integration statistics header.
  2. A tabbed section with the following tabs:

Integration statistics header

The Data Integration header shows the following statistics across all FileSourceCollections in the application. A FileSourceCollection represents a collection of files in a source system such as an Amazon S3 bucket.

Each header value includes a 24-hour count and a graph that shows historical values for the past 7 days:

  • Load Errors: Data upload or transform errors.
  • Failed Rows: Data rows that failed data processing.
  • Skipped Rows: Rows that failed at least one defined transformation condition during data processing.
  • Processed Rows: Number of successfully transformed and persisted data records.
  • Created Records: Number of new entity Type records created from source collections.
  • Updated Records: How many entity Type records were updated after processing new data.

For more information on uploading data, see Sync and Process Files.

Overview tab

The Overview tab lists the FileSourceCollections in your application. Each FileSourceCollection represents a collection of files from an external source system, such as an Amazon S3 bucket.

The list contains the following columns:

  • Source Collection: Name of the source collection.
  • State: Indicates whether the collection is processing data. Possible values are:
    • Inactive: Source collection is not processing data.
    • Processing: Source collection is processing data.
  • Queued source objects: Files queued for upload.
  • Process rate (last 10 minutes): How many data points were processed.
  • Skip rate (last 10 minutes): How many data points did not meet a transformation condition.
  • Failure rate (last 10 minutes): How many data points failed one or more transformations.
  • Last executed: The last time you processed data.
  • Process mode: Indicates whether an integration process was triggered manually or automatically.

By default, the list shows all source collections in your application. Use the search field to narrow the results by collection name, state, or process mode. You can also select one of the following categories:

  • File sources: Flat file sources (for example, CSV files).
  • Streaming sources: Data received from an external streaming source.
  • API sources: Data received from an external API call.

The Overview tab also contains a Bulk load data control. Select it to access the Bulk load data page.

Bulk load data

The Bulk load data page shows previously loaded source collection data and allows you to add more data to your application. Use the Filter panel to narrow the uploaded data list by:

  • Time range: A start or end date for the file's original upload.
  • Source Collection: The source collection used to process the file.

The Bulk load data page has two other panels:

  1. Select files to load:
  • This panel lists any previously uploaded files. You can select files here to be transformed and uploaded into your application.
  • Select the Upload Files icon (sheet with an upward-pointing arrow) to add new files from your computer to your application. The total upload size cannot exceed 2GB.

    Link new files to the correct source collection to transform them to the target Type. The data is not present in your application until you use the Load file data panel.

  1. Load selected files:
  • After you select one or more files in the 1. Select files to load panel, the Load file data button appears in this panel.
  • Select Load file data to process and upload data from the selected files into your application.
  • Select Back to return to the Data Integration Overview tab.

Volume chart tab

The Volume chart tab shows data processing statistics in a graph. The X-axis represents dates, and defaults to the last month. The Y-axis represents the number of rows processed. The graph defaults to showing the number of Processed rows in the last month. Select the Showing dropdown menu to show different statistics, such as:

  • Data load errors
  • Failed rows
  • Skipped rows
  • Created records
  • Updated records

Select the For dropdown menu to change the time resolution. Possible values are:

  • Last hour
  • Last day
  • Last week
  • Last month
  • Last year

Quality heatmap tab

The Quality heatmap tab provides a visual overview of data processed in your application over a period of time. By default, the tab shows daily aggregated statistics across the last 30 days. Double-click a date column to view detailed data for that day.

Status details tab

The Status details tab shows the status of file processing jobs. The list includes current and completed tasks. Use the search field in the list header to find files by their ID, Source Collection, or State values.

Each list item shows:

  • ID: A unique identifier generated from the processed file's name. Select this value to show additional details about the job in a new panel on the page.
  • Source Collection: The source collection used to process the file.
  • State: The status of the processing task. For error states, select this column to view error details in a window.
  • Processed rows: File rows that were successfully transformed and uploaded.
  • Skipped rows: File rows that were skipped by a conditional transformation.
  • Failed rows: File rows that failed transformation and were not uploaded.
  • Last updated: Last time this file was processed.

Use the Filter panel to find entries that were updated within a date range, or by their outcome status.

See also

Was this page helpful?