C3 AI Documentation Home

Google Drive File System Connector

Overview

The Google Drive FileSystem integration gives applications access to Google Drive through the C3 FileSystem abstraction. After you configure the integration, you use the standard file system APIs to list, read, and optionally write files in Google Drive.

The integration supports two access modes:

  • Read/write access using a Google Shared Drive
  • Read-only access to files explicitly shared with a service account

At a high level, the integration has four layers, in this order: the mount, the FileSourceSystem, the credentials, and the FileSystem operations.

Each layer depends on the previous one. Configure all four layers consistently, or the integration does not work.

You can also sync only the files that changed, with the Google Drive Changes API, and read Google-native files such as Google Docs by exporting them to a standard format.

Prerequisites

Before you configure the Google Drive FileSystem, meet the following prerequisites.

Google service account

  1. Create a service account in the Google Cloud Console.
  2. Generate a JSON key for the service account.
  3. Capture the following values from the key:
    • secretKey (private key)
    • accountId (client ID)
    • accountEmail
    • accessKey (private key ID)

Shared Drive (required for read/write mode)

  1. Create a Google Shared Drive, not a regular shared folder.
  2. Grant the service account Organizer permission on the Shared Drive. The Google Drive sharing dialog calls this role Manager.

Shared Drive ID

  • Retrieve the Shared Drive ID. Write access requires this ID.

About Google Shared Drives

A Google Shared Drive, formerly a Team Drive, is a Google Drive space for content that a team owns instead of a single user. The organization owns the files in a Shared Drive, not an individual user.

Key characteristics

  • You grant access at the drive level, not by sharing individual files or folders.
  • Files persist even if you remove individual users or service accounts.
  • Shared Drives support fine-grained permission roles:
    • Viewer
    • Contributor
    • Content manager
    • Organizer

A Shared Drive is not the same as a regular folder that you share with another user or service account. A shared folder does not provide the ownership model or write guarantees that this integration requires.

For the Google Drive FileSystem integration

  • Read/write access requires a Shared Drive.
  • Grant the service account Organizer permission on the Shared Drive.

Retrieve the Shared Drive ID

The Shared Drive ID identifies the Shared Drive. Read/write access requires the Shared Drive ID.

Use one of the following methods to retrieve the Shared Drive ID.

Option 1: Google Drive UI

  1. Open Google Drive in a browser.
  2. Go to Shared drives and select the target Shared Drive.
  3. Copy the URL from the browser address bar.

The URL looks similar to:

Text
https://drive.google.com/drive/folders/<id>

In this example, the Shared Drive ID is:

Text
<id>

Option 2: Google Drive API (optional)

If you manage drives programmatically, retrieve Shared Drive IDs with the Google Drive API (drives.list). Initial setup does not usually need this method.

Step 1: Configure the Google Drive mount

The Google Drive FileSystem requires a default mount to resolve gdrive:// URLs. The mount defines how the platform routes Google Drive paths. Configure the mount before any FileSourceSystem or file system operation.

Configure the default mount (current requirement)

Configure the default Google Drive mount programmatically. Use the GoogleDriveFileSystem configuration API to map the root path (/) to the Google Drive location you want.

Example

Python
config = c3.GoogleDriveFileSystem.inst().config()
config.setMount("/", "gdrive://TestDrive1/")

This configuration tells the FileSystem which Google Drive resource to treat as the root when it resolves file paths. In this example:

  • TestDrive1 becomes the resourceName used in all gdrive:// URLs.
  • The mount routes any path of the form gdrive://TestDrive1/<path> to Google Drive.

To verify this configuration, call the listFiles() operation in the final step of this procedure.

This verification works only after you configure the Google Drive credentials. The FileSystem needs valid credentials to authenticate and list files from the mounted drive.

Step 2: Define the FileSourceSystem (metadata)

Define the Google Drive FileSourceSystem in metadata, so that it exists before runtime. Create a JSON file in the /metadata/FileSourceSystem directory, similar to the following example.

Metadata example

JSON
{
  "type": "FileSourceSystem",
  "name": "testgdrive1",
  "rootUrlOverride": "gdrive://TestDrive1/"
}

Required fields

  • rootUrlOverride: Must match the mounted gdrive:// root. The rootUrlOverride value binds the FileSourceSystem to the mount.

Important notes

  • Create the FileSourceSystem in metadata, for example in the application repository.
  • This workflow does not support creating or upserting the FileSourceSystem at runtime.
  • If the name or rootUrlOverride does not match the configured mount, the connector does not work.

Step 3: Configure drive access mode

The driveId setting decides how the Google Drive FileSystem behaves.

Read/write mode (Shared Drive)

Read/write mode is the recommended mode, and the only mode that supports write access.

Text
driveId = "<SharedDriveId>"

When you set driveId to a Google Shared Drive ID:

  • The file system works within the Shared Drive namespace.
  • The service account must have Organizer permission on the Shared Drive.
  • Read, write, create, move, copy, and delete operations work.
  • This configuration is the only one validated for write access.
  • This configuration is the recommended enterprise setup.

Read-only mode (no Shared Drive)

Text
driveId = null

When you set driveId to null:

  • The file system does not target a specific Shared Drive.
  • The file system can access files and folders explicitly shared with the service account.
  • Listing and reading files work.
  • Write operations are not available in this mode.

Step 4: Configure credentials

Build the credentials from the service account key, then attach them to the FileSourceSystem.

Define credential parameters

Before you build GoogleDriveCredentials, define the Drive context and the service account credentials.

Define driveId

Text
driveId = "<SharedDriveId>"

  • If you use a Google Shared Drive, set driveId to the Shared Drive ID. You can copy the Shared Drive ID from the Drive URL.

  • If you do not use a Shared Drive and need only read-only access to files shared with the service account, set driveId to null (None in Python).

Define json_creds

Python
json_creds = {
    "type": "GoogleDriveCredentials",
    "secretKey": "<PRIVATE_KEY>",
    "accountId": "<CLIENT_ID>",
    "accountEmail": "<SERVICE_ACCOUNT_EMAIL>",
    "accessKey": "<PRIVATE_KEY_ID>"
}

These values come from the Google Cloud service account key for the Google Drive integration.

Construct GoogleDriveCredentials

Python
creds = c3.GoogleDriveCredentials.make(json_creds).withDriveId(driveId)

Required credential fields

  • secretKey: Service account private key
  • accountId: Client ID
  • accountEmail: Service account email
  • accessKey: Private key ID

Attach credentials to the FileSourceSystem

Python
fs = c3.FileSourceSystem.forName("testgdrive1")
fs.setCredentials(creds)

After you attach the credentials, every file system operation routed through this FileSourceSystem authenticates with them. The platform validates the credentials on the first Google Drive API call.

Step 5: Verify the integration

After you complete Steps 1 through 4, run:

Python
c3.GoogleDriveFileSystem.listFiles()

Limit listings on large drives

For folders or drives that contain many files, specify a limit when you call listFiles(), so that the call does not run for a long time.

For example:

Python
c3.GoogleDriveFileSystem.listFiles(None, 10)

A successful call confirms that:

  • The mount exists.
  • The credentials are valid.
  • The configured Drive context is accessible.

Verify read/write behavior

Write a file and read it back to check the access mode:

Python
c3.File.make("gdrive://TestDrive1/example.txt").writeString("Hello")
c3.File.make("gdrive://TestDrive1/example.txt").readString()

The expected result depends on the access mode:

  • Shared Drive mode: Both write and read operations succeed.
  • Read-only mode (driveId = null): Read operations succeed, and write operations fail.

Common failure cases

File system operations fail, because the platform cannot resolve the URL, in any of the following cases:

  • The mount does not exist.
  • The mount name does not match the FileSourceSystem resource name in the root URL.
  • The gdrive:// URL uses a resource name that does not match the configured mount.

Operational notes and guardrails

  • The platform validates credentials on the first Google Drive API call.
  • Most configuration errors surface during listFiles() or file operations.
  • The mount name and rootUrlOverride must always match.
  • Do not hard-code private keys in source code. Use secure secret management.
  • Any workflow that needs write access requires Shared Drive mode.

Incremental sync with the Google Drive Changes API

By default, the Google Drive FileSystem runs a full scan. The full scan lists all files and compares them against local state to detect new or changed content. For large drives, a full scan takes a long time even when no files have changed.

GoogleDriveFileSystem supports incremental sync with the Google Drive Changes API. After each successful sync, the platform stores a delta token. On later runs, the platform uses the token to retrieve only the files that changed, instead of scanning the full file set. A sync with no new changes finishes much sooner than a full scan.

Before you enable incremental sync, configure a working Google Drive FileSourceSystem: the mount, the FileSourceSystem metadata, and the service account credentials, as Steps 1 through 4 describe.

Enable incremental sync

Change data capture (CDC) sync applies only when you set cdcOrder="$native" on the FileSourceCollection. Collections without this setting keep the full-scan behavior.

Step 6: Opt a FileSourceCollection into CDC

Set cdcOrder = "$native" on your FileSourceCollection. This value routes every sync through changes.list instead of a full listing.

To opt in through metadata, add cdcOrder to your FileSourceCollection JSON in the /metadata/FileSourceCollection directory of your package:

JSON
{
  "name": "MyDriveCollection",
  "source": "MyDriveSourceType",
  "sourceSystem": {
    "name": "MyGoogleDriveSourceSystem"
  },
  "cdcOrder": "$native"
}

To opt in programmatically, update the collection:

JavaScript
FileSourceCollection.forName("MyDriveCollection")
  .withCdcOrder("$native")
  .upsert();

Step 7: Run the sync

Trigger the FileSourceCollection the way you normally do. On a CDC-enabled collection, each of the following entry points follows the CDC path:

  • FileSourceCollection.process(spec): the standard entry point
  • SourceFile.syncAll(fsc.inboxUrl(), spec): direct invocation against a specific inbox URL
  • SourceFile.syncAll(null, spec): fans out across all FileSourceCollections with processMode MANUAL

For example:

JavaScript
FileSourceCollection.forName("MyDriveCollection")
  .process(DataIntegSpec.make());

Step 8: Inspect the persisted checkpoint

The platform stores the delta token on the FileSourceCollection's SourceCollection.Cdc.Checkpoint. Inspect the checkpoint at any time:

JavaScript
var ckpt = FileSourceCollection.forName("MyDriveCollection").cdcCheckpoint();

ckpt.value;
// -> { "deltaToken": "<token>" }

The token advances after every successful sync. The first run seeds the token from a full listing. Each later run advances the token to the cursor that the most recent changes.list page returns. The platform persists the new token only after the dispatched FileSystemBatchJob completes every batch successfully. When a sync schedules no work because nothing changed, the platform writes the token immediately.

Access Google-native file formats in Data Integration

The Google Drive API cannot read Google-native formats, such as Google Docs, Sheets, and Slides, directly. To use these files in Data Integration (DI) pipelines, first export (transcode) them to a supported format such as TXT, CSV, or PDF.

The Google Drive API does not read the content of Docs, Sheets, or Slides. DI workflows can process the file only after you export it to a standard format.

Supported transcoding mappings

Export Google-native file formats to a supported format before you use them in Data Integration pipelines.

Google-native formatSupported export formats
Google DocsPlain text, PDF
Google SheetsCSV
Google SlidesPDF

For unsupported formats, export the file to PDF.

You can write content only to Google Docs. Writing to Google Sheets or Google Slides is not supported.

Create a Google Docs file

The following example creates a Google Docs file and shows the limits on reading Google-native formats.

Python
gfs = c3.FileSystem.gdrive()
path = "google_test/testDoc"
testfile = gfs.makeFile(path).withContentType(c3.MediaType.GOOGLE_DOCS)
createdFile = gfs.createFile(testfile)
file = gfs.openFile(createdFile)
file

You can write content to a Google Docs file with writeString(), but a direct read of the file fails by design. To access the contents, first transcode the file into a supported format, such as plain text or PDF.

Python
file.writeString("C3 rocks")
try:
    file.read()
except:
    print("File could not be read")
Text
File could not be read

Because Google Docs cannot be read directly, transcode the file into plain text first. Standard file read operations then return its contents.

Transcode the file

The following example reads the contents of a Google Docs file by exporting it to a supported format.

Python
file = file.transcode(
    c3.MediaType.PLAIN_TEXT,
    file.contentType
)

gfs.openFile(file)
file.readString()

The output shows the metadata for the new plain-text file that transcoding creates, followed by the file's contents. The GoogleDriveFile object shows that the platform converted the original Google Docs file into a text/plain file (testDoc.txt) under the same parent directory. The readString() call returns the extracted text ("C3 rocks").

Text
({
  "type" : "GoogleDriveFile",
  "contentLength" : 11,
  "contentType" : "text/plain",
  "lastModified" : "2025-06-25T00:09:25Z",
  "lastModifiedBy" : "testaccount@testproject-461317.iam.gserviceaccount.com",
  "contentMD5" : "001b17fbcaf316a88ea59646d14f820e",
  "contentSHA1" : "f91ae838dc3743df96d8e78169ead0a4864cf391",
  "hasMetadata" : true,
  "url" : "gdrive://TestDrive/google_test/testDoc.txt",
  "name" : "testDoc.txt",
  "parentId" : "1jFZUI9C0a4aI-1T8HMrLzprIGP4_M9z8",
  "fileId" : "1uoBCz6dewFmgs5R78pEzUYYd5uITlb_0"
},
"C3 rocks")

Transcoding behavior and notes

When you transcode Google-native files (Docs, Sheets, Slides), keep the following in mind:

  • Transcoding creates a new file. Transcoding does not modify the original file.
  • Original files remain until you delete them. To remove the source file, use options such as withDeleteOriginal(true).
  • PDF reads return binary data. To extract text from a PDF, use a PDF processing library.

See also

The following Google resources cover service account credentials and Google Shared Drives, which this integration depends on.

  • Create credentials for Google Workspace APIs: How to create service account credentials, generate JSON keys, and configure authentication for Google Drive APIs.
  • What are shared drives?: How Shared Drives work, how they differ from shared folders, and how Google manages permissions and ownership.
  • How file access works in shared drives: How permissions, roles, and ownership work in Google Shared Drives, including the Manager, Content manager, Contributor, Commenter, and Viewer roles. Use the page to see how a role governs access to files and folders in a Shared Drive, and how permission changes affect read and write behavior.
Was this page helpful?