C3 AI Documentation Home

Understand OneDrive and SharePoint Incremental Sync Behavior

The Microsoft Graph (OneDrive / SharePoint) FileSystem supports incremental sync using Microsoft Graph's driveItem: delta API. The platform stores a delta token after a synchronization run and uses that token during future runs to request only items that were created, modified, or deleted since the previous synchronization point. This pattern is commonly referred to as Change Data Capture (CDC).

Compared to full-drive scans:

  • No-op syncs complete significantly faster.
  • Synchronization cost scales with the size of the change set rather than the size of the drive.
  • Deleted items become detectable through the Microsoft Graph change feed.

CDC is supported at two levels:

  1. Direct FileSystem API — pass a deltaToken through ListFilesOperationSpec.
  2. Data Integration — set cdcOrder = "$native" on a FileSourceCollection and allow the platform to manage token persistence automatically.

[!NOTE] Requires a configured Microsoft Graph FileSystem, including a mount, FileSourceSystem, and valid credentials. See OneDrive and SharePoint Connectors. The same connector supports both OneDrive and SharePoint.

[!IMPORTANT] Microsoft Graph driveItem: delta queries are eventually consistent. Newly created, modified, or deleted items may not appear immediately in the change feed after an update occurs. A synchronization run immediately after a write operation may return an empty change set even when changes were made. In these cases, changes may appear during a later synchronization cycle. To reduce the likelihood of delayed updates, avoid scheduling sync runs too frequently.

Concepts

Delta Tokens

Each driveItem: delta response returns:

  • A page of changed items.
  • @odata.nextLink — pagination cursor for the current result set.
  • @odata.deltaLink — synchronization cursor for the next sync cycle.

The platform exposes the next synchronization cursor as ListFilesResult.newDeltaToken. Persist this token and provide it during the next sync request using ListFilesOperationSpec.deltaToken.

Initial vs. Delta Runs

Initial Run

When no delta token is supplied:

  • The connector performs a full listing of the configured drive or folder.
  • The platform seeds a delta token for future synchronization runs.
  • All discovered files are processed.

Delta Run

When a delta token is supplied:

  • Only changes since the previous synchronization checkpoint are returned.
  • Added and modified files are included in files.
  • Deleted items are included in deletedUrls.
  • A new delta token is returned for the next run.

Invalid Delta Tokens

Microsoft Graph may invalidate delta tokens due to expiration or backend synchronization changes.

When this occurs:

  • Direct FileSystem API workflows must re-seed the synchronization state by performing a new full listing.
  • Data Integration workflows automatically perform a full re-synchronization to establish a new checkpoint.

Direct FileSystem API

Use the Direct FileSystem API for ad-hoc scripts or custom workflows outside Data Integration.

Seed a Delta Token

JavaScript
var fs = FileSystem.fromUrlOrEncodedPath("msgraph://MySite/Documents/myFolder/");

var initial = fs.listFiles(
    "msgraph://MySite/Documents/myFolder/",
    -1,
    null,
    true,
    ListFilesOperationSpec.make()
);

var seedToken = initial.newDeltaToken();

Use the Delta Token on Subsequent Runs

JavaScript
var delta = fs.listFiles(
    "msgraph://MySite/Documents/myFolder/",
    -1,
    null,
    true,
    ListFilesOperationSpec.make().withDeltaToken(seedToken)
);

seedToken = delta.newDeltaToken();

Only advance the token after the current synchronization work completes successfully. Otherwise, failed items may not be reprocessed during the next run.

Handle Invalid Tokens

JavaScript
try {
    delta = fs.listFiles(
        url,
        -1,
        null,
        true,
        ListFilesOperationSpec.make().withDeltaToken(seedToken)
    );
} catch (e) {
    delta = fs.listFiles(
        url,
        -1,
        null,
        true,
        ListFilesOperationSpec.make()
    );
}

seedToken = delta.newDeltaToken();

Behavior Notes

  • Scope and glob filters are applied client-side, consistent with full listings.
  • OneDrive and SharePoint both use the same Microsoft Graph FileSystem implementation.
  • Setting readMetadata = true populates extended metadata such as: eTag, cTag, createdBy, lastModifiedBy, and createdDateTime.
  • Deleted items may return opaque URLs such as msgraph://<drive>#<itemId> when Microsoft Graph no longer retains the original parent path.

Data Integration via FileSourceCollection

Within Data Integration, the platform automatically handles delta token seeding, persistence, and advancement.

Enable Native CDC

Set cdcOrder = "$native" on the FileSourceCollection.

Example metadata configuration:

JSON
{
  "name": "MyOneDriveCollection",
  "source": "MyOneDriveSourceType",
  "sourceSystem": {
    "name": "MyOneDriveSourceSystem"
  },
  "cdcOrder": "$native"
}

Or configure it programmatically:

JavaScript
FileSourceCollection.forName("MyOneDriveCollection")
    .withCdcOrder("$native")
    .upsert();

Run the Synchronization

Run the pipeline using the standard Data Integration execution flow:

JavaScript
FileSourceCollection.forName("MyOneDriveCollection")
    .process(DataIntegSpec.make());

CDC-enabled collections automatically use incremental synchronization behavior.

Synchronization Behavior

Run TypeBehavior
First RunPerforms a full listing, seeds a delta token, and processes all files.
Subsequent RunsRequests and processes only changed items.
No-op RunsCompletes without additional file processing when no changes are detected.
Invalid TokenAutomatically performs a full re-synchronization to establish a new checkpoint.
Deleted ItemsDeleted items are surfaced through the Microsoft Graph change feed. Downstream cleanup or removal behavior depends on the pipeline implementation and downstream processing logic.

The file processing flow remains identical to standard file ingestion behavior. Only the file discovery mechanism changes.

Deletion Handling

Deleted items are surfaced through the Microsoft Graph change feed. If downstream systems require cleanup, archival, or deactivation behavior, implement the appropriate deletion-handling workflow within the pipeline.

Deleted items are surfaced through the Microsoft Graph change feed, but the platform does not automatically remove downstream records or indexed content associated with deleted files.

If your pipeline requires deleted content to be removed, archived, or marked inactive in downstream systems, configure a pre- or post-process hook in the pipeline to handle deletion events.

Verifying the Integration

  1. Run the initial synchronization successfully.
  2. Trigger another sync without modifying files.
  3. Confirm that no additional files are processed.
  4. Modify, add, or delete a file in the source location.
  5. Trigger another synchronization run.
  6. Confirm that only changed items are processed.

Limitations and Known Caveats

  • Microsoft Graph delta queries are eventually consistent.
  • Delta tokens are scoped to a specific drive, folder scope, and authentication context.
  • Do not reuse delta tokens across different drives, folders, or tenants.
  • When a file is deleted, Microsoft Graph includes it in the change feed's deletedUrls list. In most cases, the platform can surface the full original path of the deleted file. However, if the parent folder has also been deleted or is no longer resolvable by Graph, the platform falls back to an opaque URL in the format msgraph://<drive>#<itemId> — a reference that contains the drive identifier and the item's unique ID, but not a human-readable file path. These opaque URLs are still unique and stable per item, so they can be used to identify and act on the deleted record in downstream systems, even without the original path.
  • Items moved outside the configured synchronization scope may not appear as standard changes in subtree-scoped delta queries.

See Also

Was this page helpful?