Understand Google Drive Incremental Sync Behavior
This topic explains how incremental sync works for Google Drive file source collections using the Google Drive Changes API. It covers change detection behavior, delta token management, replay and recovery semantics, deletion handling, observability, and performance considerations for CDC-enabled sync workloads.
How Incremental Sync Works
First Run
The stored checkpoint is empty. The platform performs a full listing of the configured Drive scope, dispatches all discovered files through the standard processing path, and stores an initial delta token once all batches complete successfully.
Subsequent Runs
The platform reads the stored token, calls changes.list, and retrieves only the files that have changed or been deleted since the previous successful sync.
The change set is dispatched through FileSystemBatchJob.startForFiles(), and the new token is stored once all batches complete successfully.
No-Op Runs
When changes.list returns no files and no deletions, no batch job is scheduled. Since no file-processing work is scheduled during a no-op sync, the platform persists the newly returned delta token immediately instead of waiting for a FileSystemBatchJob completion callback. This ensures the next sync begins from the most recent checkpoint even when no files have changed.
Invalid Token
On InvalidDeltaTokenException, the platform falls back once to a full listing to re-seed the token. No data is lost; the run behaves like a fresh first run and the next run resumes incremental behavior.
What the System Detects
Incremental sync detects three categories of file changes:
- Added files are ingested as new content.
- Modified files are re-fetched and re-indexed.
- Removed or trashed files are surfaced through the Google Drive Changes API as deletions.
The platform logs deleted items during CDC syncs but does not automatically remove or tombstone the corresponding SourceFile records. If your application must react to deletions, for example, to remove downstream indexed entities, implement that behavior using pre-process or post-process hooks.
For unstructured DI pipelines, downstream indexing behavior should account for added, modified, and deleted content events. Added and modified files trigger re-ingestion and reprocessing. Deleted files are surfaced through the CDC sync path, but automatic downstream deactivation or index removal is not performed by the platform and must be implemented separately if required.
Token Persistence and Failure Semantics
The delta token is written to SourceCollection.Cdc.Checkpoint only after all scheduled batches complete successfully.
| Outcome | Token behavior |
|---|---|
| All batches succeed | Token advances once, after all work has been applied |
| Any batch fails | Token is not advanced; next sync replays the same delta window |
| No-changes sync | No batch job is scheduled; token is advanced immediately |
Successfully processed files in a prior run are treated as no-ops on replay, so retrying a failed sync is safe.
Token Invalidation and Fallback
A stored delta token may become invalid if it has expired or if a significant change was made to the Drive configuration. When this occurs, the platform automatically falls back to a full listing to re-seed the token, after which incremental sync resumes normally.
[!NOTE] Auto-recovery applies only when using the Data Integration path. If you are using the direct FileSystem API, your code must catch
InvalidDeltaTokenExceptionand re-issue the call without adeltaTokento obtain a fresh seed token.
If you encounter unexpected behavior during an incremental sync, look for an InvalidDeltaTokenException in the sync logs. The platform raises this exception when Drive returns HTTP 410 (Gone) or 404 for an expired or invalid token.
Sync Observability
Each CDC sync run logs the following details for monitoring and troubleshooting:
- Counts of added, updated, deleted, skipped, and failed items processed during the sync
- The previous delta token used for the sync and the newly persisted delta token
For Data Integration collections, you can also inspect the persisted checkpoint directly:
var ckpt = FileSourceCollection.forName("MyDriveCollection").cdcCheckpoint();
ckpt.value();Verify the delta token advances after each successful sync to ensure the CDC is active and consuming the change stream correctly.
Performance Considerations
When working with folders or drives that contain a large number of files, consider specifying a limit when calling listFiles() to avoid long-running operations:
c3.GoogleDriveFileSystem.listFiles(None, 10)For recurring sync workloads, use incremental sync (see Incremental Sync with the Google Drive Changes API) instead of relying on listFiles() for change detection. Full scans can take up to 15 minutes for large drives with no new content; incremental sync reduces this to seconds by querying only changed files since the last stored token.