C3 AI Documentation Home

Configure Kafka as a Streaming Data Source

An Apache Kafka Topic connector enables ingestion of streaming data from a Kafka topic into the platform. This connector is categorized under Cloud message connectors and is used for event-driven or real-time data integration scenarios.

Kafka connectors allow the platform to consume messages published to a topic and make them available for downstream processing, transformation, or storage.

When to Use a Kafka Topic Connector

Use this connector when:

  • Data is produced as a continuous stream of events
  • You need near real-time ingestion
  • Your source system publishes data to Kafka topics

Typical use cases include:

  • Event-driven pipelines
  • Log and telemetry ingestion
  • Streaming analytics

Connector Configuration Model

The Kafka Topic connector is configured using the Data Connector wizard, which separates configuration into distinct stages:

StagePurpose
ConnectorSelect the connector type
DestinationDefine where data is written
AuthenticationConfigure secure access to Kafka
ReviewValidate and save the connector

Authentication Model

Kafka uses API key-based authentication. During connector setup, you provide the following credentials in the Auth step:

  • API Key — API key from your Kafka cloud provider for authenticating to this cluster.
  • API Secret — API secret paired with the key above.

Both fields are required and are stored securely as part of the connector configuration.

Advanced Configuration

Advanced settings are optional and typically only required for:

  • Private or self-managed Kafka clusters
  • Custom TLS configurations

Most standard cloud Kafka deployments do not require additional configuration.

Task: Connect to Apache Kafka Topic

Before You Begin

Before configuring the connector, ensure that:

  • Ensure you have access to the Kafka broker and topic.
  • Ensure required credentials are available, or be prepared to create them.
  • Have the Kafka topic name you want to connect to.
  • If creating new credentials: have access to your Kafka provider's admin console to generate API keys.

Step 1: Select Connector Type

On the New Connector page:

  1. From the left navigation sidebar, go to Data → Data Fusion. This opens the Data Integration canvas where your pipeline nodes are displayed.
  2. Click the + Add Data Source button in the top-right toolbar of the canvas. This opens the Configure Connector dialog.
  3. In the connector selection screen, you will see connectors organized by category. Scroll down to the Cloud Message Brokers / Streaming section and click Kafka. You can also filter by clicking the "Cloud Message Brokers / Streaming (3)" tag, or search for "Kafka" in the search bar.
  4. After selecting Kafka, you are taken to the Destination step.

[!TIP] Use the Cloud message filter to narrow the list to messaging connectors.

Step 2: Configure Destination

On the Destination page, provide the connector endpoint and location details.

FieldDescription
Connector name (required)Unique name for the connector (for example, Production Kafka Topic).
Description (optional)Brief description of the connector.
Broker Endpoint (required)Bootstrap broker URL from your managed Kafka cluster. Use the bootstrap endpoint your provider lists for SASL-secured clients.

Click Continue.

Step 3: Configure Authentication

On the Authentication page:

Enter your Kafka API credentials:

  • API Key (required) — API key from your Kafka cloud provider for authenticating to this cluster.
  • API Secret (required) — API secret paired with the key above.

Step 4: Review and Test Configuration

On the Review & Test page:

  1. Review the connector summary, including:
    • Connector name
    • Connector type
    • Endpoint configuration
  2. Expand the Destination section to verify the configured Kafka bootstrap endpoint.
  3. Confirm that the endpoint value matches your Kafka cluster endpoint.
  4. Click Next to continue.

[!NOTE] The current prototype review page displays only a subset of connector configuration values. Additional stream-specific configuration options will be introduced in a future release.

Auto-Populate Connector Configuration (Optional)

Instead of manually entering connector values, you can use the Auto-populate workflow to extract configuration values from JSON or a JDBC URL and automatically populate connector fields.

The auto-populate workflow supports:

  • Extracting configuration keys from existing connection configurations
  • Suggesting mappings to connector fields
  • Automatically populating values across the connector wizard

Open the Auto-Populate Workflow

From the connector wizard:

  1. Click Auto-populate in the left navigation panel.

The Auto-Map Fields dialog opens.

Provide Configuration Input

Under Provide Input, select one of the following input methods:

Input TypeDescription
JSONPaste structured configuration data in JSON format.
JDBC URLPaste a JDBC connection string.

You can:

  • Paste configuration values directly
  • Upload a file containing the configuration

Extract and Review Mappings

  1. Paste the configuration into the input area.
  2. Click Extract & Suggest Mapping.

The system:

  • Extracts configuration keys from the provided input
  • Suggests mappings to connector fields across the wizard

Under Map Extracted Keys to Connector Fields, review the suggested mappings and adjust them if needed.

Note: Only configuration keys are used for mapping. Sensitive values remain private.

Apply Mappings

  1. Click Apply Mapping & Fill Fields.

The wizard automatically populates matching values across:

  • Destination
  • Authentication
  • Advanced options

Review the populated values and update any fields as needed.

Continue Connector Configuration

After applying mappings:

  1. Return to the connector wizard.
  2. Verify the populated values.
  3. Complete any remaining required fields.
  4. Continue with connector validation and testing.

Step 5: Configure Source Collection

After clicking Next on the Review & Test step, the wizard advances to the Configure Source Collection step. It contains:

  • Name (required): A unique name for this source collection. Must be alphanumeric and not begin with a number.
  • Description (optional): A description for this source collection.
  • Stream selected section:
    • Stream: Enter the name of the Kafka stream (topic) to ingest data from. A data preview will appear on the right once a valid stream name is entered.
    • Partition:
      • All — consume from all partitions (default).
      • Enter partition — specify a particular partition number.
  • Click Save to complete the source collection configuration.

Step 6: Connect to the Pipeline

Once the Kafka source node is configured, connect it to downstream nodes in the pipeline.

Next Steps

  • Configure transformations and mappings
  • Run and monitor ingestion jobs

Streaming platforms such as Kafka provide real-time event data as it is generated. Kafka is commonly used for messaging, telemetry ingestion, activity tracking, and stream processing.

In Data Fusion, Kafka enables continuous ingestion of event-driven data for use cases such as:

  • Real-time ingestion — Process telemetry or event streams from applications and devices
  • AI-driven workflows — Support near real-time inference and decision-making
  • Operational monitoring — Power dashboards and alerting systems with up-to-date data

Troubleshoot Kafka connections

Unable to list available streams

If the system displays “Unable to list available streams”:

  • Enter the Kafka topic manually in the Broker name field
  • Verify that the Kafka credentials are valid and correctly entered
  • Confirm that the Kafka credentials have DESCRIBE permission on the topic

No partitions available

If the Partition (preview only) dropdown is empty:

  • Confirm that the Kafka credentials have DESCRIBE permission
  • Verify that the topic exists in the Kafka cluster

No preview data displayed

Data is retrieved only after selecting a partition.

If no data appears after selecting a partition:

  • Confirm that the Kafka credentials have READ permission
  • Verify that the selected partition contains data
  • Ensure the topic name and partition are entered correctly

If the request succeeds but returns no data, the UI displays No data available.

If the request fails, an error is displayed, which may indicate:

  • Invalid credentials
  • Incorrect topic or partition

Connection fails during setup

If the connector fails to validate:

  • Verify the endpoint (bootstrap server) is correct
  • Confirm that the API key and API secret are valid and active
  • Ensure network access to the Kafka cluster is available

Key considerations

  • Kafka topics must exist in the external Kafka system; Data Fusion does not create topics
  • Topic discovery may not be available depending on permissions
  • Data preview requires selecting a partition and depends on both access permissions and data availability

See also

Was this page helpful?