C3 AI Documentation Home

Custom Connector Framework

What Is the Custom Connector Framework?

The Custom Connector Framework lets the C3 platform connect to external systems not covered by the platform's built-in SQL, file, cloud-message, or API source types—for example, SaaS REST APIs, proprietary protocols, or other externally accessible data sources.

A single connector implementation can support any combination of:

  • Live queries
  • Data Integration (DI) ingestion—non-incremental or incremental (CDC)
  • Write-back to the external system

For REST-backed systems, VirtualRestSourceSystem extends VirtualSourceSystem with common REST capabilities, including request execution and pagination, while the connector author implements the API-specific behavior. For more information, see Implement a REST-Based Connector.

Architecture and Components

A connector implementation contains the logic that communicates with the external system, including query translation and execution. The platform exposes that single implementation through several component types, each covering a different entry point:

ComponentRole
VirtualSourceSystemThe connector implementation. Its element type R is Obj or Arrow. Every connector implements ping. The other hooks are optional: readStream, readTupleStream, batches, singleBatch, countObjs, writeBatch, writeStream, removeBatch, removeStream, deleteAll, and toWriteSpec. The connector has no separate DI read hook.
VirtualPersistableMixed into an application type to route fetch, evaluate, fetchCount, and supported write operations to the configured VirtualSourceSystem through @connector(sourceSystem=...).
VirtualSourceCollectionOwns DI batching and checkpoint state. It splits the read into batches and reads each batch through the connector's readStream or readTupleStream hook.
VirtualTargetCollectionConnects the connector to the DI write pipeline. It prefers writeStream for pipeline output, and uses writeBatch when writeStream is not implemented.
VirtualRestSourceSystemA specialized VirtualSourceSystem for REST APIs that provides common REST functionality while the connector author implements the API-specific hooks.

Custom Connector High-level Architecture

readStream returns rows as the connector's element type R. R is Obj or Arrow. For a live call, the platform bridges Obj and Arrow when the caller needs the other representation. A STATELESS DI read can use a connector that binds R=Arrow. A WATERMARK or CUSTOM read accumulates the checkpoint per row. That read requires R=Obj and readStream, or a readTupleStream implementation. Arrow is not used for those incremental reads.

The framework allows the same connector implementation to be reused across these entry points while providing supporting capabilities such as authentication and, where applicable, REST pagination, DI batching, and checkpoint management for incremental ingestion.

Data.Spark.Planner (@beta) plans distributed reads of connector-backed types. It partitions a VirtualPersistable read through the connector's batches.

Access Patterns: Live Query, DI Ingestion, and Write-Back

  • Live query: An on-demand read of a connector-backed type. The application calls fetch, evaluate, or fetchCount. The platform resolves the type's VirtualSourceSystem and queries the external system directly, outside the DI pipeline. fetch, fetchObjStream, and fetchArrowStream call readStream. evaluate calls readTupleStream. fetchCount and exists call countObjs. The call fails as unsupported when the required hook is not implemented.
  • DI ingestion: A pipeline-driven read through a VirtualSourceCollection. The extraction strategy is STATELESS, WATERMARK, or CUSTOM. DI reads are batched live reads. Each batch ends in the same readStream or readTupleStream hook a live call uses. The collection builds the batches and the checkpoint state. Retrieved data is streamed into the pipeline, or staged first, depending on configuration.
  • Write-back: A write from the platform to the external system. Live writes are application operations on the connector-backed type. DI writes are the load stage of a pipeline, through a VirtualTargetCollection. Both use writeBatch and writeStream for writes, and removeBatch and removeStream for removes. A live write prefers the batch hook. A DI write prefers the stream hook. Live removeAll calls deleteAll.

A single connector can support any combination of these access patterns.

Using Different Schemas for Live Queries and DI Ingestion

A connector-backed application type defines an external type—the schema used for live queries and write operations. Separately, a VirtualSourceCollection defines a source type—the schema used for DI ingestion.

These schemas can differ, allowing the same connector to expose a simplified schema for interactive queries while the DI pipeline ingests a richer schema optimized for transformation and loading.

Mapping C3 Names to the External Schema

Set @connector(table=...) on the type when the external table name differs from the C3 type name. Set @connector(fieldName=...) on a field when the external field name differs from the C3 field name. Both default to the C3 name when unset.

The platform resolves table and passes it on the connector read spec and write spec. For a VirtualRestSourceSystem, table is the REST query path. That path selects the @rest method that reads the type.

The connector reads @connector(fieldName=...) from the field. The platform does not resolve that name. Sample connectors share the lookup in SampleFieldNames, through externalName, externalToFieldMap, and fieldNamed.

Credential Model

Credentials are configured per VirtualSourceSystem instance rather than embedded in the connector implementation. This allows the same connector type to be configured with different credentials across environments or deployments.

Checkpoint and Batching Model

For incremental DI, checkpoint state lives on the VirtualSourceCollection.

For WATERMARK extraction, the collection builds the checkpoint from its watermark fields.

For CUSTOM extraction, the collection implements these four hooks:

  • batchesForCustom encodes the incremental cursor on each batch.
  • generateCdcCheckpointFields seeds the accumulator from the prior checkpoint.
  • updateCheckpointFields folds each read row into the accumulator.
  • mergeCheckpointAccumulators merges the per-batch accumulators.

The connector has no CDC hook beyond its ordinary read hook. The platform persists the merged accumulator's field map as the checkpoint, and only after the run succeeds.

A DI run dispatches its batches through VirtualSourceCollection.ReadBatchJob. The job reads batches across the cluster. Each batch writes its accumulator to a VirtualSourceCollection.ReadBatchResult row keyed by job and batch. A retry of that batch overwrites the same row. A second run of the same collection is rejected while a lease row for that collection is held.

The job persists the merged checkpoint only after every batch succeeds. It then removes staged files. If a batch fails, the checkpoint stays at the previous value. The next run resumes from that checkpoint.

A collection can split the read into batches that the external system can process independently. Job options limit how many of those batches run at the same time. A connector that implements only singleBatch is read as one batch.

Was this page helpful?