Loading Historical Data

C3 AI Application data models often contain calculated fields and auto-computed data. When new data is ingested, these calculated fields are automatically invalidated and placed in the InvalidationQueue for recomputation.

Copy link to this section

Understand the impact of historical data loads

For incremental (recurring) data loads, this automatic invalidation ensures that the application remains up to date without user intervention. However, during historical (one-time) data loads, this process can lead to inefficiencies, as it may result in hundreds of thousands of duplicate invalidated entries in the queues.

Calculated fields are only one form of auto-computed data in the C3 AI Platform. Applications can also include normalized timeseries, denormalized hierarchies, and other analytics that would become invalid. Refresh these analytics when adding new data.

Consider a calculated field on an ItemFacility for current inventory, which derives from the latest daily snapshot of inventory. Loading 900 days of inventory snapshots may cause the field to be recalculated 900 times, even though only the most recent date is needed after all snapshots are ingested. This issue is magnified with thousands of parts, multiple calculated fields, and large volumes of data.

Copy link to this section

Disable asynchronous processing during historical data load

To optimize historical data loading, temporarily disable the async processing feature to prevent unnecessary recalculations.

InvalidationConfig.inst().withUpsertAsyncProcessingDisabled(true).setConfig()

This command temporarily disables async processing, allowing you to efficiently load historical data.

Copy link to this section

Re-enable asynchronous processing after data load

Once the historical data load is complete, re-enable async processing by running the following command:

InvalidationConfig.inst().withUpsertAsyncProcessingDisabled(false).setConfig()

Copy link to this section

Perform an application rebuild

After re-enabling async processing, it is possible to manually refresh any calculated data with PersistableRefreshable#refreshCalcFields to ensure computed data is current. However, performing this step manually is not recommended, as it can be difficult to account for any data dependencies between calculated fields.

The C3 Agentic AI Platform provides a job, AppRebuilder, that can re-compute all calculated fields and data for an application. A simple job that re-computes calculated fields can be started by running the following command:

App.rebuild(spec={storedCalcs: true})

Important: App.rebuild requires a minimum of 2 compute threads to proceed.

Then, a job can be monitored or cancelled with the following commands:

// Checks the status of the AppRebuilder job
App.rebuildStatus()

// Cancels the AppRebuilder job
App.cancelRebuild()

The spec passed into App#rebuild is an AppRebuildSpec. Use c3ShowType(AppRebuildSpec) to explore additional parameters that can be passed into the rebuild specification.