C3 AI Documentation Home

Incremental Sync for OneDrive and SharePoint (Change Data Capture)

Overview

The Microsoft Graph (OneDrive / SharePoint) FileSystem supports incremental sync based on Graph's driveItem: delta API. The platform persists a delta link (Graph's @odata.deltaLink) and on the next run asks Graph only for the items created, modified, or removed since that link. This pattern is commonly called Change Data Capture (CDC).

Compared to a full re-list:

  • No-op syncs finish in seconds (one HTTP call instead of paginating the drive).
  • Cost scales with the change set, not the drive size.
  • Deletions become observable via Graph's deleted facet.

CDC is exposed at two layers:

  1. Direct FileSystem API — pass deltaToken on ListFilesOperationSpec.
  2. Data Integration — set cdcOrder = "$native" on a FileSourceCollection; the platform persists and advances the token automatically.

Requires a working Microsoft Graph FileSystem (mount, FileSourceSystem, credentials). See OneDrive and SharePoint Connectors. The same connector covers both OneDrive and SharePoint; CDC behaves identically against both.

Graph's driveItem: delta is eventually consistent. A write may take seconds to minutes to surface in the change feed, and Graph's listing index and change feed propagate independently. A delta sync run immediately after a write may legitimately return an empty change set; the event will surface on a later run. No events are lost — only their delivery may be delayed across runs. Schedule syncs no faster than ~1–2 minutes apart for the change feed to settle.

Concepts

Delta token

Each driveItem: delta page returns:

  • A page of driveItem objects (created/modified items, plus removed items carrying a deleted facet).
  • @odata.nextLink — pagination cursor for the same result set.
  • @odata.deltaLink — cursor for the next sync (only set on the final page).

The platform exposes the next-sync cursor as ListFilesResult.newDeltaToken. Persist it and pass it back as ListFilesOperationSpec.deltaToken on the next call.

Initial vs. delta runs

  • Initial run (deltaToken empty): a full listing, plus a lightweight delta(token='latest') call to seed newDeltaToken for the next run.
  • Delta run (deltaToken set): replays the supplied delta link and returns:
    • files — created or modified items.
    • deletedUrls — URLs of removed items. Falls back to msgraph://<drive>#<itemId> when Graph has pruned the parent path.
    • newDeltaToken — cursor for the next run.

Invalid delta tokens

Graph rejects expired tokens with HTTP 410 (resyncRequired), or in some tenants HTTP 404 referencing the delta token. The FileSystem surfaces this as a typed InvalidDeltaTokenException.

  • Direct API: catch and re-issue the call with no deltaToken to re-seed.
  • Data Integration: the platform handles this automatically — it falls back once to a full listing and persists the new token.

Direct FileSystem API

Use this for ad-hoc scripts or custom workflows outside Data Integration.

1. Seed a token via a full listing

JavaScript
var fs = FileSystem.fromUrlOrEncodedPath("msgraph://MySite/Documents/myFolder/");
var initial = fs.listFiles("msgraph://MySite/Documents/myFolder/", -1, null, true,
                           ListFilesOperationSpec.make());
var seedToken = initial.newDeltaToken(); // persist somewhere durable

2. Pass the token on subsequent runs

JavaScript
var delta = fs.listFiles("msgraph://MySite/Documents/myFolder/", -1, null, true,
                         ListFilesOperationSpec.make().withDeltaToken(seedToken));
// delta.files() / delta.deletedUrls() / delta.newDeltaToken()
seedToken = delta.newDeltaToken(); // persist for next run

Only advance the token after the work for the current delta succeeds; otherwise failed items will not be reprocessed.

3. Handle invalid tokens

JavaScript
try {
   delta = fs.listFiles(url, -1, null, true,
                        ListFilesOperationSpec.make().withDeltaToken(seedToken));
} catch (e) {
   if (e instanceof InvalidDeltaTokenException) {
      delta = fs.listFiles(url, -1, null, true, ListFilesOperationSpec.make());
   } else { throw e; }
}
seedToken = delta.newDeltaToken();

Behavior notes

  • Scope and glob filters are applied client-side, just like for full listings.
  • OneDrive and SharePoint both use the same MsGraphFileSystem; the drive label in the URL determines the target.
  • readMetadata = true populates extended metadata (eTag, cTag, createdBy, lastModifiedBy, createdDateTime, …).
  • Deletions are best-effort: opaque msgraph://<drive>#<itemId> is returned when Graph has pruned the deleted item's parent path.

Data Integration via FileSourceCollection

The platform handles seeding, persisting, and advancing the delta token. The change set flows through the same FileSystemBatchJob infrastructure as the full-list path.

1. Opt the FileSourceCollection into native CDC

Set cdcOrder = "$native". Either via metadata:

JSON
{
  "name": "MyOneDriveCollection",
  "source": "MyOneDriveSourceType",
  "sourceSystem": { "name": "MyOneDriveSourceSystem" },
  "cdcOrder": "$native"
}

…or programmatically:

JavaScript
FileSourceCollection.forName("MyOneDriveCollection")
                    .withCdcOrder("$native").upsert();

2. Run the sync

Trigger any of the standard entry points; CDC-enabled collections automatically take the CDC path:

JavaScript
FileSourceCollection.forName("MyOneDriveCollection").process(DataIntegSpec.make());

3. Inspect the persisted checkpoint

JavaScript
FileSourceCollection.forName("MyOneDriveCollection").cdcCheckpoint().value();
// -> { "deltaToken": "<delta link>" }

The token advances after every successful sync.

Behavior summary

Run typeBehavior
FirstFull listing → seeds token → dispatches all files to a batch job.
SubsequentCalls driveItem: delta → dispatches only the change set.
No-opNo batch job scheduled. Token still advances.
Invalid tokenFalls back once to a full listing (which re-seeds the token).

The change set is dispatched via FileSystemBatchJob.startForFiles(...), so existing batch tuning (sizes, priorities, retries) applies unchanged. Only the discovery step changes; processing is identical.

Deletions

The platform logs the deletion count on each CDC run but does not automatically remove SourceFile records. To act on deletions, implement your own pre/post-process hook against the logged count and the delta token.

Verifying the integration

  1. Trigger one sync; confirm cdcCheckpoint().value() is non-null.
  2. Wait ≥1 minute, trigger again; confirm zero new FileSystemBatchJob entries (no-op).
  3. Update or delete a file, wait, trigger again; confirm deltaToken has advanced.

Limitations and known caveats

  • Deletions are not auto-acted-upon (see Deletions).
  • Opaque deletion URLs (msgraph://<drive>#<itemId>) when Graph has pruned the parent path; the itemId is still unique and stable.
  • Eventual consistency of Graph's listing index and change feed (see the Important callout).
  • Token portability: delta links are scoped to one drive/library and one service principal. Do not transplant across resources.
  • Move-out semantics: items moved out of the scoped subtree between two delta calls are filtered out client-side. To react to such moves, scope the delta to the drive root.

See also

Was this page helpful?