ETL from Streaming Sources
ETL (Extract, Transform, Load) from streaming sources enables you to process real-time data efficiently, allowing for timely insights and analytics. This capability is crucial for applications that require immediate data availability and responsiveness.
There are two ways to connect streaming sources to C3 Agentic AI Platform:
- Data Fusion — a low-code tool for building data pipelines in C3 AI Studio
- C3 Type System — a solution that requires more code and offers more in-depth functionality
Data Fusion is in Beta. Please contact your C3 AI representative to enable this feature.
Example connections
The following connectors are examples of cloud brokers and streaming sources you can use for your data pipelines.
By utilizing these connectors, you can harness real-time data for analytics and decision-making. This integration not only improves data accessibility but also supports the development of responsive applications that can adapt to changing data landscapes.
Apache Kafka connector
The Apache Kafka Connector facilitates the integration of Kafka streams into the C3 Agentic AI Platform. It allows users to extract data from Kafka topics, transform it as needed, and load it into C3 AI's data models. This connector supports high-throughput data ingestion, making it suitable for applications that handle large volumes of streaming data. It also leverages Kafka's robust messaging capabilities to ensure reliable data delivery.
Amazon Kinesis connector
The Amazon Kinesis Connector provides a seamless way to ingest data from Amazon Kinesis streams into C3 AI. This connector is designed for real-time data processing, enabling users to capture and analyze data from various sources, such as IoT devices and web applications. With Kinesis, you can scale your data ingestion processes to accommodate fluctuating data volumes, ensuring that insights are derived promptly.
Azure Event Hubs connector
The Azure Event Hubs Connector allows for the integration of data from Azure Event Hubs into the C3 Agentic AI Platform. This connector is optimized for high-throughput scenarios, enabling users to process millions of events per second. It supports real-time analytics and event-driven architectures, making it ideal for applications that require immediate responses to incoming data streams.
Override configurations for streaming sources
Use the CloudMessageSourceCollection.Config instance in CloudMessageSourceCollection to override streaming source preset values. For example, to override the SourceCollection to accept AVRO formatting, use the following code snippet, using your source collection name in place of <source_collection_name>:
var config = CloudMessageSourceCollection.forName("<source_collection_name>").config();
config.withContentTypeOverride(MediaType.AVRO).setConfig();