C3 AI Documentation Home

Connect BigQuery with a Service Account

Connecting BigQuery with a service account involves using the service account credentials to authenticate and authorize access to BigQuery. This allows C3 AI applications to interact with BigQuery without requiring individual user credentials.

C3 AI Studio offers a low-code tool for building data pipelines called Data Fusion. Data Fusion configures data sources and connects them to C3 Canonicals and Entities. For more information, refer to Data Fusion Overview.

Quick start - connect Google BigQuery using a service account

This section provides the fastest path to validate connectivity between C3 AI and Google BigQuery using a service account. It assumes you already have a Google service account JSON key and want to quickly confirm access before proceeding with full configuration.

Prerequisites for quick start

  • A Google Cloud service account JSON key with access to BigQuery
  • A SqlSourceSystem for BigQuery already defined in application metadata
    (If the SqlSourceSystem does not exist, define it in metadata before proceeding. Credential configuration does not create source systems.)
  • Access to C3 AI Studio (console or notebook)

Quick start steps

Run the following in C3 AI Studio (for example, in the console or a configuration notebook):

JavaScript
// Load the service account JSON securely
var serviceAccountJson = "<SERVICE_ACCOUNT_JSON>";

// Build the BigQuery JDBC URL with OAuthJWT parameters
var jdbcUrl =
  "jdbc:googlebigquery:" +
  "AuthScheme=OAuthJWT;" +
  "OAuthJWTCertType=GOOGLEJSONBLOB;" +
  "ProjectId=<GCP_PROJECT_ID>;" +
  "DatasetId=<BIGQUERY_DATASET>;" +
  "OAuthJWTCert=" + serviceAccountJson + ";" +
  "OAuthJWTSubject=<SERVICE_ACCOUNT_EMAIL>";

// Construct JDBC credentials
var creds = c3.JdbcCredentials.make({
  url: jdbcUrl
});

// Attach credentials to the existing SqlSourceSystem
var sqlSystem = c3.SqlSourceSystem.forName("BigQuerySourceSystem");
sqlSystem.setCredentials(creds);

// Validate connectivity
sqlSystem.connect().listTableNames();

Expected result

  • A list of BigQuery table names is returned.
  • The service account authentication and JDBC configuration are valid. If this succeeds, the BigQuery connector is correctly configured and ready for use by Source Collections and ingestion pipelines.

If this fails

Verify the following:

  • The service account has BigQuery Data Viewer and BigQuery Job User roles.
  • The project ID, dataset ID, and service account email are correct.
  • The service account JSON key is valid and securely loaded.

Next steps

After completing this Quick Start, continue with the rest of this topic for:

  • Secure credential storage guidance
  • Parameter explanations
  • Operational best practices
  • Troubleshooting scenarios

Connect Google BigQuery using a service account

This topic explains how to connect Google BigQuery to C3 AI using a service account and the SqlSourceSystem. Service account authentication enables secure, non-interactive access to BigQuery for automated data integration pipelines.

This approach is recommended for production deployments and aligns with current connector security and configuration practices.

Overview

In C3 AI, Google BigQuery is configured as an external SQL data source using a SqlSourceSystem. Authentication is performed using OAuth 2.0 with JSON Web Tokens (OAuthJWT) backed by a Google service account.

At a high level, the configuration flow is:

Service Account → JDBC Credentials → SqlSourceSystem → Source Collections → Pipelines

The service account is used to authenticate JDBC connections to BigQuery. Once credentials are attached to the SqlSourceSystem, downstream Source Collections and pipelines reuse the same authenticated connection.

Prerequisites

Before configuring the BigQuery connector, ensure the following:

  • Access to the Google Cloud Console

  • A Google Cloud project with BigQuery enabled

  • Permission to create and manage service accounts

  • The SqlSourceSystem and Source Collections for Google BigQuery are already defined via application metadata. Credential configuration does not create source systems, tables, or source collections.

    The steps in this topic assume these metadata objects already exist and focus only on securely attaching credentials and validating connectivity. Deploy the metadata to your development environment so the platform is aware of the source system and source collections.

  • Access to C3 AI Studio (console, Data Fusion, or a notebook environment)

Create a google service account

  1. Open the Google Cloud Console.
  2. Navigate to IAM & Admin → Service Accounts.
  3. Click Create Service Account.
  4. Provide a name and description for the service account.
  5. Assign the required roles, for example:
    • BigQuery Data Viewer
    • BigQuery Job User
  6. Generate a JSON key for the service account.
  7. Download and securely store the JSON key file.

The service account JSON key contains sensitive credentials and must be stored securely.
Do not commit it to source control or inline it directly in code.

Securely store the service account credentials

Store the service account JSON key using your organization’s secure secret management approach (for example, Vault or environment-specific secret storage).

JavaScript
var serviceAccountJson = "<SERVICE_ACCOUNT_JSON>";

The placeholder represents credentials loaded securely at runtime, not hardcoded values.

Construct JDBC credentials using OAuthJWT

C3 AI connects to Google BigQuery over JDBC using OAuthJWT authentication. Username and password authentication is not supported.

In this configuration, OAuthJWT parameters are supplied directly in the BigQuery JDBC connection string. The JDBC driver uses these parameters to generate signed JWTs and obtain access tokens from Google on behalf of the service account.

Run this step in C3 AI Studio (for example, in the console or a configuration notebook) as part of external source system setup. This code executes outside of pipelines and transforms and must be completed before configuring Source Collections.

JavaScript
var jdbcUrl =
  "jdbc:googlebigquery:" +
  "AuthScheme=OAuthJWT;" +
  "OAuthJWTCertType=GOOGLEJSONBLOB;" +
  "ProjectId=<GCP_PROJECT_ID>;" +
  "DatasetId=<BIGQUERY_DATASET>;" +
  "OAuthJWTCert=" + serviceAccountJson + ";" +
  "OAuthJWTSubject=<SERVICE_ACCOUNT_EMAIL>";

var creds = c3.JdbcCredentials.make({
  url: jdbcUrl
});

Parameter details

  • AuthScheme
    Specifies OAuth 2.0 authentication using JSON Web Tokens (OAuthJWT).

  • OAuthJWTCertType
    Indicates that credentials are provided as a Google service account JSON object.

  • OAuthJWTCert
    Contains the service account key material used to sign JWTs and request access tokens.

  • ProjectId / DatasetId
    Define the BigQuery project and dataset context.

  • OAuthJWTSubject
    The service account email used as the token subject.

The service account JSON must be loaded securely at runtime.
Do not hardcode credentials or commit service account keys to source control.

Attach credentials to the SqlSourceSystem

Retrieve the existing SqlSourceSystem and attach the credentials.

JavaScript
var sqlSystem = c3.SqlSourceSystem.forName("TestExternalSystem2");
sqlSystem.setCredentials(creds);

The SqlSourceSystem represents the authoritative connection point for BigQuery and is used by downstream Source Collections and pipelines. The SqlSourceSystem must already exist (for example, defined in application metadata) before credentials are attached.

Validate the connection

Verify that the connection is working by listing the available tables.

JavaScript
sqlSystem.connect().listTableNames();

Expected result

  • A list of table names is returned

  • The connection and authentication are correctly configured

If errors occur, validate:

  • Service account permissions

  • JDBC URL correctness

  • Secure storage and retrieval of credentials

Optional - Verify BigQuery connectivity using an external entity

After configuring the BigQuery SqlSourceSystem, you can optionally verify the connection by defining an external entity that maps to a BigQuery table. This confirms that authentication, schema discovery, and query execution are functioning correctly.

The following example maps a BigQuery table to a C3 AI entity using external storage:

Type
entity type IrisBigQuerySP mixes External, NoSystemCols
  schema name "`dev.APPARATUS`" {

  @db(dataTypeOverride='string')
  id: ~ schema name "APPARATUSID"

  retirementdatetime:        datetime no tz schema name 'RETIREMENTDATETIME'
  firstinstallationdatetime: datetime no tz schema name 'FIRSTINSTALLATIONDATETIME'
  servicepointid:            string   schema name 'SERVICEPOINTID'
}

If this entity compiles successfully and returns results when queried, the BigQuery connection is correctly configured.

This entity definition is provided only as a verification example.
It is not required to configure the BigQuery connector and is typically defined as part of application data modeling or Source Collection setup.

Operational notes and best practices

  • Always configure BigQuery connections through the SqlSourceSystem.
  • Do not set credentials directly on JdbcStore.
  • Do not inline service account JSON in source code.
  • Use secure secret management for credential storage.

Troubleshooting

Common error - Permission or authentication failure

Cause

  • Missing BigQuery roles on the service account
  • Incorrect or expired service account key
  • Incorrect JDBC URL

Resolution

  • Verify service account roles in Google Cloud
  • Re-generate the service account key if needed
  • Confirm the JDBC URL matches your BigQuery project and dataset

See also

Was this page helpful?