C3 AI Documentation Home

Implement a REST-Based Connector

A REST-based connector enables the C3 AI Platform to communicate with external systems that expose REST APIs.

Instead of implementing HTTP communication and pagination logic from scratch, create the connector by mixing VirtualRestSourceSystem. The platform provides common REST infrastructure, including request execution, pagination, authentication, retries, connection pooling, and timeout handling. You implement the API-specific behavior and can optionally override the default request execution when additional customization is required.

VirtualRestSourceSystem

VirtualRestSourceSystem mixes VirtualSourceSystem with built-in support for REST-based integrations.

The framework is responsible for:

  • Executing HTTP requests.
  • Managing authentication, including OAuth configuration.
  • Driving the pagination loop.
  • Converting returned records into platform objects.
  • Supporting both live query operations and Data Integration reads.

The connector author implements the following extension hooks:

HookPurpose
buildArgsBuilds the request arguments for the REST API call.
extractRecordsExtracts records from the API response.
hasNextPageDetermines whether another page of data is available.
nextPageStateReturns the state required to retrieve the next page.
pingVerifies connectivity to the external system.

Note: VirtualRestSourceSystem does not provide a default implementation of evaluateTupleStream. A connector that supports live-query evaluate() operations, such as aggregations, must implement this hook itself; otherwise the platform fails with an error when evaluate() is invoked. AggregatingRestConnector is an example of a connector that opts in by implementing evaluateTupleStream to translate the evaluate specification into its own API's native aggregation query.

Create a REST Connector

Create a REST connector by defining a subtype of VirtualRestSourceSystem.

The connector specifies:

  • A configuration type.
  • A credential type. See Implement a Custom Connector for how to define one.
  • The REST extension hooks required by the connector.

Example

Type
type FakeOffsetRestConnector mixes VirtualRestSourceSystem<
    SampleRestConnectorConfig,
    SampleConnectorCredentials> {

    ping:           ~ js-server
    buildArgs:      ~ js-server
    extractRecords: ~ js-server
    hasNextPage:    ~ js-server
    nextPageState:  ~ js-server
}

type SampleRestConnectorConfig mixes VirtualRestSourceSystem.Config<SampleConnectorCredentials>

You can extend the configuration type with connector-specific settings in addition to the REST configuration provided by the framework.

Configure Authentication

VirtualRestSourceSystem supports a static Authorization header value and OAuth 2.0 client credentials through its RestConfig-derived settings.

For a static header, set the auth field on the connector's configuration. The platform sends that value unchanged as the Authorization header. A bearer token must already include the Bearer prefix and the following space. This value is a secret field (@config(secret=true)).

Configure OAuth through oauth2Config. OAuth uses the client-credentials grant with a private-key JWT assertion signed with RS256. When oauth2Config is set, the platform uses that token instead of the auth value.

Configure Retries

By default, the platform retries an idempotent request, such as a GET, once after HTTP 503 or 429. A POST request is not retried by default. To retry more than once, or on other codes such as 500, 502, and 504, set retries and retryHttpCodes. A code in retryHttpCodes is retried for every request method, including POST.

Define REST API Methods

Define one or more REST methods by annotating connector methods with @rest.

Bind the source type to a connector instance with @connector(sourceSystem, table). sourceSystem names the connector instance that backs the type, and table names the REST query path. The platform routes requests to the @rest method whose name equals table. If no method name matches, it uses the method whose @rest(uri=...) equals table. If you omit table, the platform uses the name of the source type.

Note: table is a general-purpose field on @connector. Storage-style connectors use it to name the external collection or table. For a VirtualRestSourceSystem connector, it identifies the REST query path that selects the @rest method.

Example

Type
type RealRestConnector mixes VirtualRestSourceSystem<
    SampleRestConnectorConfig,
    SampleConnectorCredentials> {

    @rest(uri='widgets', method='GET')
    widgets: abstract member function(offset: any): json
}

@connector(sourceSystem='RealRestConnector', table='widgets')
entity type RestWidgetSource mixes Source {

    id:   ~
    name: string
}

The REST method returns the raw response from the external API. The connector extracts the records from the response by implementing extractRecords.

Note: The table value must equal the name or the uri of exactly one @rest method on the bound connector. If no method matches, the read fails with an error that lists the tables the connector can resolve. If more than one method matches, the read fails and asks you to name one method or give its full uri.

Pagination

Many REST APIs return data in multiple pages. VirtualRestSourceSystem provides a generic pagination framework that repeatedly invokes the configured REST method until all available records have been retrieved.

For each request, the framework:

  1. Calls buildArgs to prepare the request arguments.
  2. Invokes the configured REST method.
  3. Calls extractRecords to retrieve the records from the response.
  4. Calls hasNextPage to determine whether another page is available.
  5. Calls nextPageState to prepare the next request.

The same pagination framework is used for both live query operations and Data Integration reads. Implement the pagination hooks according to the pagination model used by the external API.

Implement buildArgs

Implement buildArgs to construct the request arguments passed to the REST method.

The framework calls this method before each REST request and passes the current page state, if one exists. Use the method to translate the query specification into the arguments required by the external API.

During Data Integration reads, the behavior depends on the configured extraction strategy:

Extraction strategybuildArgs behavior
WATERMARKReceives the incremental filter generated by the framework through spec.filter(). Translate this filter into the corresponding query parameters required by the external API.
STATELESSReceives the configured read scope, such as filters, limits, or ordering. No checkpoint-based incremental filter is generated.
CUSTOMReceives the read scope whose filter already encodes the connector-owned cursor, built by the source collection's batchesForCustom. buildArgs runs the same as for the other strategies; it does not need separate handling for the custom checkpoint.

Example

JavaScript
function buildArgs(type, spec, pageState) {
    var offset = pageState == null ? 0 : pageState;
    return [offset];
}

The implementation can also translate filters, limits, and ordering into the request parameters required by the external API.

Implement extractRecords

Implement extractRecords to extract the records returned by the REST API.

The framework passes the raw response from the REST method to this hook. Return only the collection of records that should be processed by the platform.

Example

JavaScript
function extractRecords(response) {
    if (!response || !response.records) {
        return [];
    }

    return response.records;
}

The framework converts the returned records into platform objects.

Implement hasNextPage

Implement hasNextPage to determine whether another page of results should be requested. The implementation depends on the pagination model used by the external API.

Cursor-based pagination

JavaScript
function hasNextPage(response, fetchedSoFar, limit) {
    return !!(response && response.nextPage != null);
}

Offset-based pagination

JavaScript
function hasNextPage(response, fetchedSoFar, limit) {
    if (!response) {
        return false;
    }

    return response.offset + PAGE_SIZE < response.total;
}

Implement nextPageState

Implement nextPageState to provide the information required to retrieve the next page of results.

The returned value becomes the page state passed to the next invocation of buildArgs.

Cursor-based pagination

JavaScript
function nextPageState(response, currentPageState, limit) {
    return response.nextPage;
}

Offset-based pagination

JavaScript
function nextPageState(response, currentPageState, limit) {
    var offset = currentPageState == null ? 0 : currentPageState;
    return offset + PAGE_SIZE;
}

The page state can represent a cursor, an offset, or any value required by the external API's pagination model.

Override invokeApi (Optional)

VirtualRestSourceSystem provides a default implementation for executing REST requests.

Use the standard extension hooks (buildArgs, extractRecords, hasNextPage, and nextPageState) whenever they satisfy your connector's requirements. If additional customization of the request execution process is required, override invokeApi().

Implement ping

Implement ping to verify connectivity to the external REST API.

The platform uses this method to determine whether the configured endpoint is reachable.

Example

JavaScript
function ping() {
    return PingResult.make({
        url: this.config().url,
        reachable: true
    });
}

In a production connector, implement ping by issuing a lightweight request against the configured REST endpoint.

Was this page helpful?