C3 AI Documentation Home

Incremental Sync for Google Drive (Change Data Capture)

Overview

The Google Drive FileSystem integration supports incremental sync based on Google Drive's changes.list API. Instead of re-scanning every file in a Drive on every sync, the platform can persist a delta token (Drive's startPageToken) and on the next run ask Drive only for the files that have been created, modified, or removed since that token. This pattern is commonly called Change Data Capture (CDC). The delta token naming follows the prevailing industry vocabulary for "resume a listing where a prior one left off, returning only what changed" — most notably Microsoft Graph / OData's deltaToken / deltaLink; provider-specific equivalents include Drive's own startPageToken, Dropbox's cursor, and Box's stream_position.

The benefits compared to a full re-list on every run are:

  • No-op syncs complete in seconds rather than minutes. A run that finds nothing changed performs one short HTTP call instead of paginating the full Drive listing.
  • Network and quota usage scale with the change set, not the Drive size. Drives with millions of files but only a few daily updates do not pay the full-list cost on each run.
  • Deletions become observable. Drive's changes.list reports removed and trashed files, which a static listing cannot.

The same delta-token plumbing is exposed at two layers:

  1. Direct FileSystem API — pass deltaToken on ListFilesOperationSpec to GoogleDriveFileSystem.listFiles(...) to drive your own incremental workflow.
  2. Data Integration — opt a FileSourceCollection into native CDC by setting cdcOrder = "$native". The platform reads, advances, and persists the token automatically per source collection in SourceCollection.Cdc.Checkpoint, and dispatches the change set through the same distributed batch pipeline that the legacy full-list path uses.

Incremental sync requires the standard Google Drive FileSystem to be configured first, including the mount, FileSourceSystem, and service-account credentials. See Google Drive File System Connector for the prerequisites and one-time setup steps.

Drive's changes.list API is eventually consistent. A successful write to Drive does not immediately appear in the change log — propagation can take anywhere from a few seconds to several minutes, and Drive's listing index and change log propagate independently of one another. This means:

  • A delta sync triggered immediately after a Drive write may legitimately return an empty change set. The missed event will surface on a subsequent sync once Drive's change log catches up; no events are lost, only their delivery may be delayed across runs.
  • "Token has advanced" does not imply "every prior write has been observed". Both pieces of state are advanced as the provider sees them, not as the user committed them.
  • For workflows that require a write to surface on the very next sync, schedule syncs no faster than the provider's typical propagation delay (a 1–2 minute floor is a reasonable starting point) rather than triggering back-to-back syncs after each write.

This is a Google Drive characteristic, not a platform behavior. The same caveat applies whether you use the direct FileSystem API or the Data Integration path. See Limitations and known caveats for related notes.

Concepts

Delta token

Drive's changes.list endpoint is paginated by an opaque page token. Each call returns:

  • changes — files created, modified, removed, or trashed since the token.
  • nextPageToken — the cursor to use for the next page of the same result set.
  • newStartPageToken — the cursor to use for the next sync (i.e., everything that has changed after this call).

The platform exposes the cursor used for the next sync on the result of any listFiles call as ListFilesResult.newDeltaToken. You persist that value somewhere safe (the platform does this for you in the FileSourceCollection path) and pass it back as ListFilesOperationSpec.deltaToken on the following call.

Initial run vs. delta run

  • Initial run — deltaToken is empty/null. The FileSystem does a full listing (the same listing the API has always returned) but additionally stamps a fresh newDeltaToken on the way out, so the next call can switch to the incremental path.
  • Delta run — deltaToken is set. The FileSystem calls changes.list, walks every page, applies any client-side scope/glob filters, and returns:
    • files — created or modified files in the change window, with metadata populated.
    • deletedUrls — URLs of files that were removed or trashed since the token. Drive surfaces deletions as either a fully-resolved gdrive://... URL when the deleted file's parent path is still cached, or as an opaque gdrive://<resource>#<fileId> fragment when the parent has been pruned.
    • newDeltaToken — the cursor to use for the next delta run.

Invalid delta tokens

Drive will reject a token that is too old or otherwise no longer valid (HTTP 410 "Gone" or 404 "Invalid / expired page token"). The FileSystem surfaces this as a typed InvalidDeltaTokenException.

When this exception is thrown:

  • In the direct FileSystem API, your code should catch it and re-issue the call with no deltaToken to obtain a fresh seed token via a full listing.
  • In the Data Integration path, the platform handles this for you automatically: on InvalidDeltaTokenException it falls back once to a full listing (which re-seeds the token) and persists the new token onto the SourceCollection.Cdc.Checkpoint.

Using deltaToken directly via the FileSystem API

Use this approach when you want to drive your own incremental workflow over Drive without going through Data Integration — for example, an ad-hoc script, a custom batch job, or a cron action that operates on Drive content outside the FileSourceCollection model.

Step 1: Issue an initial full listing to obtain a seed token

JavaScript
var fs = FileSystem.fromUrlOrEncodedPath("gdrive://TestDrive/myFolder/");
var initial = fs.listFiles("gdrive://TestDrive/myFolder/", -1, null, true,
                           ListFilesOperationSpec.make());
// initial.files()        -> the full listing (same as before this feature existed)
// initial.newDeltaToken() -> the seed token to use on the next call
var seedToken = initial.newDeltaToken();

initial.newDeltaToken() is non-null whenever the listing has been fully consumed (i.e. there is no nextMarker left). Persist seedToken somewhere durable.

Step 2: On subsequent runs, pass the stored token

JavaScript
var delta = fs.listFiles("gdrive://TestDrive/myFolder/", -1, null, true,
                         ListFilesOperationSpec.make().withDeltaToken(seedToken));
// delta.files()        -> only files created or modified since seedToken
// delta.deletedUrls()  -> URLs of files removed or trashed since seedToken
// delta.newDeltaToken() -> token to use on the *next* call
seedToken = delta.newDeltaToken(); // persist for next run

Apply your own action to each file in delta.files() and (optionally) each URL in delta.deletedUrls(), then persist delta.newDeltaToken() for the next iteration. The token must only be advanced after the work for the current delta has been acknowledged; if you advance the token before processing succeeds, files in the failed batch will not be reprocessed on the next run.

Step 3: Handle invalid tokens

JavaScript
try {
   delta = fs.listFiles(url, -1, null, true,
                        ListFilesOperationSpec.make().withDeltaToken(seedToken));
} catch (e) {
   if (e instanceof InvalidDeltaTokenException) {
      // Re-seed via a full listing.
      delta = fs.listFiles(url, -1, null, true, ListFilesOperationSpec.make());
   } else {
      throw e;
   }
}
seedToken = delta.newDeltaToken();

Behavior notes

  • Scope and glob filters are applied client-side. Drive's changes.list does not accept server-side path or glob filters, so the FileSystem prunes the change set after receiving it. The same urlOrEncodedPath prefix and globPattern you would have used for a full listing are honored on the delta.
  • Shared drives are supported. supportsAllDrives and includeItemsFromAllDrives are set automatically when the configured driveId is non-null.
  • readMetadata = true populates file size and checksums. When you need the size, md5Checksum, or sha1Checksum of changed files (for example, to compare against your own state), pass readMetadata = true (the fourth argument of listFiles).
  • Deletions are best-effort. If Drive can no longer resolve a deleted file's parent path, the URL falls back to gdrive://<resource>#<fileId>. This is a known limitation of the Drive Changes API for fully-deleted files.

Using CDC in Data Integration via FileSourceCollection

Use this approach when Google Drive content is being ingested through the standard Data Integration pipeline. The platform handles seeding, persisting, and advancing the delta token for you, and dispatches the change set through the same distributed batch infrastructure (FileSystemBatchJob) that the full-list path has always used.

Prerequisites

You must have a working Google Drive FileSourceSystem — the mount, FileSourceSystem metadata, and service-account credentials are described in Google Drive File System Connector. Incremental sync layers on top of an existing FileSourceSystem; it does not replace any of the one-time setup.

Step 1: Opt a FileSourceCollection into native CDC

Set cdcOrder = "$native" on your FileSourceCollection. This sentinel string is the public convention that opts a file source into filesystem-native change streams; for the Google Drive FileSystem, it routes every sync through changes.list instead of a full listing.

You can set cdcOrder either via metadata or programmatically.

Via metadata

Add cdcOrder to your FileSourceCollection JSON in the /metadata/FileSourceCollection directory of your package:

JSON
{
  "name": "MyDriveCollection",
  "source": "MyDriveSourceType",
  "sourceSystem": { "name": "MyGoogleDriveSourceSystem" },
  "cdcOrder": "$native"
}

Programmatically

JavaScript
FileSourceCollection.forName("MyDriveCollection")
                    .withCdcOrder("$native")
                    .upsert();

Once cdcOrder is set, the next sync of this collection follows the CDC path.

Step 2: Run the sync

Trigger the FileSourceCollection the same way you always have. Any of the following entry points will go through the CDC path on a CDC-enabled FileSourceCollection:

  • FileSourceCollection.process(spec) — the standard "process this collection" entry point. Internally forwards inboxUrl() to SourceFile.syncAll(...).
  • SourceFile.syncAll(fsc.inboxUrl(), spec) — direct invocation against a specific inbox URL.
  • SourceFile.syncAll(null, spec) — fans out across every FileSourceCollection whose processMode is MANUAL; CDC-enabled collections among them follow the CDC path.
JavaScript
FileSourceCollection.forName("MyDriveCollection").process(DataIntegSpec.make());

Step 3: Inspect the persisted checkpoint

The delta token is persisted on the FileSourceCollection's Cdc.Checkpoint:

JavaScript
var ckpt = FileSourceCollection.forName("MyDriveCollection").cdcCheckpoint();
ckpt.value(); // -> { "deltaToken": "<token>" }

The token advances after every successful sync. The first run seeds it from a full listing; every subsequent run advances it to the cursor returned by the most recent changes.list page. The new token is only persisted after the dispatched FileSystemBatchJob has completed every batch successfully (see "Token persistence and failure semantics" below) — when the sync schedules no work (no changes), the token is written immediately because there is nothing to fail.

Behavior

  • First run: The stored checkpoint is empty. The platform issues a full listing to enumerate the inbox and stamps a seed token onto the result. Every file in the inbox is dispatched to a FileSystemBatchJob (just like the legacy path), and the seed token is stamped onto the job and persisted from the job's allComplete once every batch processes successfully.
  • Subsequent runs: The platform reads the stored token, calls changes.list, and receives only the changed files (plus any deletions). The change set is dispatched to a FileSystemBatchJob via FileSystemBatchJob.startForFiles(...), which chunks the files into per-batch payloads and schedules them directly on the caller; the job's doStart then no-ops (no redundant second listing) and the cluster runs processBatch as usual. The new token is stamped onto the job entity and persisted from allComplete after every batch processed successfully.
  • No-op runs: When changes.list returns no files and no deletions, the platform does not schedule a batch job at all. The token is advanced immediately (there is no job that could fail) so the next run starts from a quiet baseline.
  • Invalid token: On InvalidDeltaTokenException, the platform falls back once to a full listing (which re-seeds the token). No data is lost; this run behaves like a fresh "first run" and the next run resumes incremental behavior.

Token persistence and failure semantics

The new delta-token is stamped onto the dispatched FileSystemBatchJob's cdcSourceCollectionName + cdcNewDeltaToken fields and is only written to the SourceCollection.Cdc.Checkpoint from FileSystemBatchJob#allComplete. The batch engine invokes allComplete only when every scheduled batch processes successfully; on partial failure the engine invokes failed instead and the checkpoint is not advanced.

In practice that means:

  • Happy path (all batches succeed): the token advances exactly once per sync, after the work has been applied across the cluster.
  • Any batch fails: the token stays at its prior value. The next sync's changes.list will replay the same delta window, re-dispatching the failed deltas (the FileSystemBatchJob_MethodsBase#processBatch path is idempotent at the per-SourceFile level, so successfully-applied files in the prior run are no-ops).
  • No-changes sync: no batch job is scheduled, so the new token is persisted inline. There is nothing that can fail, and skipping the immediate write would mean the FSC re-queries the same empty window forever.

This success-only persistence means a CDC-enabled collection is safe to retry on arbitrary failures — the next sync resumes from the last successfully-applied delta, not from an over-advanced cursor that would silently skip past unprocessed changes.

Distributed processing

The change set is dispatched through the same FileSystemBatchJob infrastructure the non-CDC path uses, so you get the same multi-node parallel processBatch fan-out and the same BatchFileOperationSpec knobs (batch size, priority, parallelism). The only difference is that FileSystemBatchJob.startForFiles(...) does the chunking + batch scheduling up front on the caller, so the job's doStart no-ops instead of re-listing the entire inbox — and the full change-set never has to live on the job row.

This means existing Data Integration tuning (batch sizes, queue priorities, retries) applies unchanged to CDC-enabled collections. Only the discovery step changes; the processing step is identical.

Deletions

The Drive Changes API surfaces removed and trashed files via deletedUrls. The platform currently logs the count of deletions on each CDC run but does not automatically remove or tombstone the corresponding SourceFile records. If your application needs to react to deletions (for example, to remove downstream entities), implement the action in your own pre/post-process hook based on the logged count and the delta token state.

Acting on deletions automatically is intentionally out of scope for the initial release. The CDC path advances the delta token whether or not deletions are surfaced, so a future enhancement that adds delete-side actions will not require any delta-token migration.

Verifying the integration

After enabling CDC, you can verify it is active by inspecting the persisted checkpoint and the queue activity:

  1. Trigger one sync. Confirm FileSourceCollection.cdcCheckpoint().value() is non-null.
  2. Wait at least a minute for Drive's change log to settle (see the eventual-consistency callout in Overview), then trigger a second sync. Confirm the BatchQueue produces zero new FileSystemBatchJob entries for this collection (the CDC path schedules nothing on a no-op delta; the legacy path would still schedule one job).
  3. Update or delete a file in the Drive inbox, wait again for Drive's change log to surface the event, then trigger another sync. Confirm the Cdc.Checkpoint.value().deltaToken field has advanced. If the token did not advance on the first try, retry after another minute — Drive's change log can lag committed writes.

Performance and quota

  • No-op runs issue exactly one changes.list HTTP call (one Drive API quota unit) regardless of inbox size.
  • Delta runs scale with the size of the change set, not the Drive size. Drives with millions of files but only a handful of daily updates pay roughly the same per-run cost as Drives with hundreds of files.
  • Initial runs still require a full listing. There is no avoiding this on the first sync of a CDC-enabled collection — the seed token can only be obtained after a full enumeration.
  • Drive quota: changes.list and changes.getStartPageToken count against your Google Cloud project's Drive API quota. Both are inexpensive (one quota unit per call).

Limitations and known caveats

  • Deletions are not auto-acted-upon. See Deletions above.
  • Deleted-file URLs may be opaque. gdrive://<resource>#<fileId> is returned when Drive can no longer resolve the deleted file's parent path. The fileId is still unique and stable.
  • Eventual consistency of changes.list. Drive's listing index and its change log are independently eventually consistent and can each lag a successful write by seconds-to- minutes. A delta sync run immediately after a write may return an empty change set; the event surfaces on a subsequent run once Drive's change log catches up. The persisted delta token still advances on every run — no events are lost, only their delivery may be delayed across runs. This is a Drive characteristic, not a platform behavior; the same caveat applies to the direct listFiles(... deltaToken ...) API. See the Important callout above for how to schedule sync runs in light of this.
  • Token portability. Delta tokens are scoped to the originating Drive (or shared Drive) and the originating service account's view. Do not attempt to transplant a token across resources.

See also

Was this page helpful?