Understand CDC Sync Modes and Staging Behavior
Sync Mode Options
The following sync modes are available when configuring CDC in C3 AI for a table. By default, Change Data Capture (CDC) supports two sync modes for relational data sources:
- Full sync
- Incremental sync
Full Sync
Loads the entire table from the source during each sync run.
This mode is available only for tables that fall within the supported small-table threshold. Full sync is suitable for:
- Initial data loads
- Small tables with low update frequency
- Tables that do not contain a reliable
last_modifiedor equivalent timestamp column
Incremental Sync
Captures only new or updated records since the previous sync checkpoint.
Incremental sync requires the source table to contain a reliable last_modified or equivalent monotonically increasing column. This mode is recommended for large or frequently updated tables.
Full Sync for Small Tables
To protect system performance and prevent resource exhaustion, the platform restricts full sync to small tables only.
When you configure CDC for a table and select full sync mode, the platform automatically validates the table size before allowing the configuration to be saved. If the table exceeds the supported size threshold, the full sync option is unavailable.
How Table Size Is Evaluated
Before enabling full sync, the platform evaluates the size of the target table using the following properties:
- Row count — the number of rows currently in the table.
- Column count — the number of columns in the table.
Full sync is gated by a row count threshold with a default maximum of 100,000 rows. This value is user configurable.
What Happens When a Table Grows Past the Threshold
Full sync eligibility is revalidated during each sync run, not only during initial configuration.
If a table grows beyond the supported threshold after full sync has already been configured, the sync job fails and the system returns an error indicating that the table exceeds the supported size limits. In this case, you must switch the table to incremental sync mode.
Manual syncs respect the configured sync mode and are subject to rate limiting. If a sync request is rejected because of rate limiting, the system returns a message indicating when the next sync is allowed.
The platform prevents concurrent sync execution for the same table. If a sync is already running, additional sync requests for that table are blocked until the active sync completes.
CDC Without Staging
Overview
In previous releases, configuring Change Data Capture (CDC) for SQL source systems required an intermediate staging layer. Captured change data was first written to staged files before it was processed by the downstream Data Integration (DI) pipeline.
With this release, CDC can run directly against supported SQL source systems without requiring an intermediate staging step. This simplifies CDC configuration and reduces the storage and operational overhead associated with managing staged files.
What Staging Is
In staging-based CDC workflows, captured SQL change data is written to an intermediate file storage layer before downstream processing occurs.
The staging layer consists of:
- A
FileSourceSystem - A
FileSourceCollection
Captured records are persisted as staged parquet files that can later be consumed by the DI pipeline.
Why Staging Was Previously Required
Earlier CDC implementations relied on staged files to:
- Persist captured change data between sync runs
- Maintain processing checkpoints
- Support replay and recovery workflows
- Decouple source ingestion from downstream processing
Why Staging Is Now Optional
The platform can now directly process CDC data from supported SQL source systems without first persisting captured records to intermediate staged files.
This streamlined execution model reduces configuration complexity and is suitable for simpler CDC workflows that do not require intermediate staged file retention or reprocessing of previously captured data.
When to Use CDC Without Staging
The following table describes common scenarios and the recommended CDC configuration approach.
| Scenario | Recommended Approach |
|---|---|
| Simple, direct CDC pipelines | Without staging |
| Minimal setup and lower storage overhead | Without staging |
| Reuse captured data across multiple transforms or pipelines | With staging |
| Replay or audit raw captured data | With staging |
Prerequisites
Before configuring CDC without staging:
- A supported SQL source system must already be configured.
- The source entity must contain a reliable
cdcOrderfield or another monotonically increasing timestamp or ordering column. - The source collection configuration must have
stage: falseset.
When stage: false is configured, CDC records are processed directly without intermediate file staging. There is no separate configuration field required to enable CDC without staging.
CDC without staging is not supported on earlier backend versions. Attempting to run pipelines configured without staging on unsupported server versions may result in pipeline execution failures.
Configure CDC Without Staging
- Configuring CDC without staging follows the standard CDC workflow, but does not require staged file configuration.
- To configure CDC without staging:
- Configure the SQL source system.
- Define the source entity type.
- Configure the entity with a
cdcOrderfield or another reliable monotonically increasing timestamp or ordering column. - Configure the
SqlSourceCollection. - Set
stage: falsein the source collection configuration. This enables CDC processing without intermediate staging. - Enable CDC using
cdcOrder, but do not configure staging-related options such as: a..withStage(true)b..withStageFileSourceCollectionName(...) - Define the target entity type and any required transforms.
- Configure the CDC processing schedule.
- Start CDC processing.
[!NOTE] In the Data Fusion UI, CDC without staging is configured through the Virtual Table workflow rather than the Load Data ingestion workflow. This allows CDC records to be processed directly without intermediate staged file ingestion.
Checkpoint Behavior Without Staging
Checkpointing continues to function in no-staging mode.
The platform maintains a CDC checkpoint to track the most recently processed record.
During each run:
- The checkpoint value is read before processing begins.
- Only records with values greater than the stored checkpoint are fetched and processed.
- The checkpoint is updated after successful processing completes.
Unlike staging-based CDC, no intermediate staged files are retained for replay or recovery.
Limitations and Considerations
When CDC is configured without staging:
- Raw captured data is not persisted to intermediate storage.
- Replay from staged files is not available.
- Recovery behavior differs from staging-based CDC workflows.
If downstream processing fails after records are fetched but before processing completes successfully, the checkpoint may need to be manually reset before rerunning the pipeline.
CDC without staging is suitable for workflows that prioritize simplified configuration and lower storage overhead, such as:
- Real-time analytics
- Operational synchronization workflows
- Lightweight ingestion pipelines
CDC with staging is better suited for workflows that require:
- Auditability
- Replay support
- Raw data retention
- Reuse of captured data across multiple downstream pipelines
Example Workflow
The following example demonstrates a simplified CDC configuration without staging.
Unlike staging-based CDC workflows, this configuration does not require:
- A
FileSourceSystem - A
FileSourceCollection - Staging-specific configuration methods
The pipeline directly processes captured SQL CDC records from the source system into the downstream DI workflow.