C3 AI Documentation Home

Data Ingestion Best Practices

Data ingestion in the C3 AI Platform generates execution metadata (for example, SourceStatus, TransformStatus, and TargetStatus) with each pipeline run. Over time, this metadata can accumulate and impact database performance if not managed proactively.

There is no automatic cleanup of ingestion metadata. Application owners are responsible for defining retention policies and determining when to archive or remove older records. Retention should be determined based on application-specific factors such as data volume and throughput.

Over time, old statuses and source files should be cleared to prevent accumulation and performance degradation.

Archiving and deleting metadata

For production environments, metadata should be archived before deletion when required for retention or audit purposes.

A typical workflow is:

  1. Export metadata – Use C3 AI Export jobs on relevant types (for example, DataIntegStatus) to extract metadata records. See Export.c3typ for details on configuring export jobs.
  2. Store exported data – Write exported data to the C3 file system or external storage (such as Amazon S3) for retention.
  3. Validate exported data (optional) – Confirm that required records have been successfully archived.
  4. Delete old metadata – Remove outdated records using platform APIs (for example, removeAll()) or scheduled cleanup jobs.

Before running a removeAll() on any table, it is best practice to test the filter statement in a fetch() or fetchCount() command first to ensure it returns the records you intend to delete.

JavaScript
// View DataIntegStatus created before '2020-10-01'
var startDate = DateTime.make('2020-10-01');
c3Grid(DataIntegStatus.fetch({
  filter: Filter.lt('meta.created', startDate)
}));
JavaScript
// Count all DataIntegStatus created before '2020-10-01'
var startDate = DateTime.make('2020-10-01');
DataIntegStatus.fetchCount({
  filter: Filter.lt('meta.created', startDate)
})

Example to delete entries older than a certain date:

JavaScript
// Removes all DataIntegStatus created before '2020-10-01'
var startDate = DateTime.make('2020-10-01');
DataIntegStatus.removeAll({
  filter: Filter.lt('meta.created', startDate)
}, true);

// Removes all SourceStatus created before '2020-10-01'
var startDate = DateTime.make('2020-10-01');
SourceStatus.removeAll({
  filter: Filter.lt('meta.created', startDate)
}, true);

Data Fusion does not provide UI controls for archiving or deleting metadata. These operations must be performed using platform-level jobs, APIs, or workflows.

Platform-specific considerations

PostgreSQL

In C3 AI, the three key data integration metadata types are SourceStatus, TransformStatus, and TargetStatus. All extend the DataIntegStatus type, which means they are stored in a single PostgreSQL table.

Monitor table size using fetchCount(). Export and remove older records when thresholds are exceeded to prevent performance degradation.

JavaScript
var count = DataIntegStatus.fetchCount();
console.log(count)

Cassandra

If you are using Cassandra as the key-value store for your C3 AI application, it is essential to adhere to best practices to ensure optimal performance.

For large volumes of data in Cassandra, you can enable cold storage to move unused data to the file system instead of retaining it in the primary store. This behavior can be configured using annotations in the type metadata.

Steps to view Cassandra ring usage in Grafana

To determine the percentage of disk usage in your Cassandra ring using the Grafana dashboard, follow these steps:

  1. Log in to Grafana.
  2. Locate the relevant dashboard for your Cassandra metrics.
  3. Check for panels displaying disk usage data. Ensure that the dashboard is configured correctly to show metrics from your Cassandra cluster.
  4. Visualize the disk usage to gain insights into the fullness of your ring. If needed, create custom queries to display the required metrics.

Automating metadata cleanup

To maintain a consistent cleanup cadence, you can automate metadata management using scheduled jobs:

  • Create a Type method that deletes records older than a retention period defined by the application (for example, based on data volume and throughput).
  • Schedule the method using a CronJob.
  • Optionally chain the cleanup to run after an Export job using a workflow.
JavaScript
// In <pkg>/src/MetadataDeleter.c3typ
type MetadataDeleter {
  deleteOldObjs: function() js-server
}

// In <pkg>/src/MetadataDeleter.js
function deleteOldObjs() {
  var startDate = DateTime.now().plusDays(-180);
  DataIntegStatus.removeAll({
    filter: Filter.lt('meta.created', startDate)
  }, true);
  return;
}
JSON
// In <pkg>/seed/CronJob/scheduledMetadataDeleter.json
{
  "id": "scheduledMetadataDeleter",
  "name": "scheduledMetadataDeleter",
  "description": "Deletes all DataIntegStatus older than 180 days. Runs on the first of each month at midnight UTC.",
  "action": {
    "typeName": "MetadataDeleter",
    "actionName": "deleteOldObjs"
  },
  "inactive": false,
  "concurrent": false,
  "runOnLeader": true,
  "trackHistory": true,
  "scheduleDef": {
    "cronExpression": "0 0 0 1 * ? *",
    "skipOverdue": true
  }
}

The retention period of 180 days used in this example is illustrative. There is no recommended default; the appropriate value depends on application-specific factors such as data volume and pipeline throughput. Adjust the value passed to plusDays() to match your application's requirements.

To run this code after an Export job in production, you can utilize a {@link WorkFlow} to schedule this type method to run after the Export job has completed.

Automation helps ensure that metadata does not accumulate to levels that impact system performance.

Conclusion

Effective metadata management is critical for maintaining the performance and stability of C3 AI applications. By defining appropriate retention strategies, archiving metadata when required, and automating cleanup processes, you can prevent performance degradation and ensure long-term system health.

See also

Was this page helpful?