Specify a KV Store
The C3 Agentic AI Platform separates its persistence layer into two primary data storage solutions: SQL store implemented with Java Database Connectivity (JBDC) and KV store (Key-Value store).
Data store solutions in the C3 Agentic AI Platform
The following table describes the KV store and SQL store solutions, their advantages and disadvantages:
| Data Store | Description | Advantages | Disadvantages |
|---|---|---|---|
| KV Store | Data storage that persists data as a set of unique identifiers. | Allows for quick data retrieval and scalable storage for your C3 AI applications. | Designed for append-only use and is not optimized for updating or removing data. |
| SQL store (default solution) | A relational database that uses JDBC (Java Database Connectivity) that you can optionally configure to use row-based storage without a structured schema. | Typically less expensive than the average KV Store, and suits small to medium amounts of data that support relational querying patterns. | Slower performance than KV Store. |
Your C3 AI environment and data store configuration determines the data storage solution for your Types.
Default databases
C3 AI recommends the following options and manages these data storage solutions to optimize performance and cost. The default database depends on your environment configuration:
| Environment | SQL store database | KV store database |
|---|---|---|
| Single node environments (SNEs) | H2 | H2 |
| SNEs with a shared database and shared environments (SEs) | PostgreSQL | Cassandra |
If you do not specify a data storage solution, the platform uses the SQL store database by default. If you use the Ann.Db annotation datastore="kv" on an entity Type, the platform uses the recommended KV store solution as described in the table.
To learn more about PostgreSQL and Cassandra databases, see the following documentation
About using a shared database
When an environment uses a shared database, multiple applications use the same database in the datastore. The applications share tables and implicit filtering on application code.
A datastore can have multiple databases, which applications may or may not share. All applications in a single node environment (SNE) share the same datastore. SNEs use unique databases per environment, while shared environments (SEs) use unique databases per application.
KV Store solutions that you can specify
Although C3 AI recommends using the default databases described in the previous section, you can specify a KV store database that is not SQL store or the default environment-based solution. The following table describes the KV store alternatives and their cost and performance considerations.
| Alternative data storage | Ann.Db @db Annotation | Cost and performance considerations |
|---|---|---|
| Cassandra | datastore="cassandra" | BASE-compliant KV store option that prioritizes availability and performance. Supports large data loads and production-level needs and is the most expensive solution. Not for relational query patterns. For smaller data loads and testing, use default SQL store instead. |
| H2 and PostgreSQL as SqlKvStore | datastore="sqlkvstore" | Non-ACID-compliant SqlKvStore databases that are middle-cost options. Optimal for smaller data loads, querying, and testing. |
| File system | datastore="fs" or datastore="kvstore.filesystem" | Non-ACID-compliant and the least expensive option. Use when you do not require complex querying, such as logging and managing raw data. For better querying, use a default data SQL store or KV store. |
| DynamoDB or Bigtable | datastore="dyndb" or datastore="KvStore.BigTable" | Not optimized for the C3 Agentic AI Platform and the provider plan determines cost. Supported as alternatives if your organization requires it, but consider other solutions for most use cases. |
For an ACID-compliant solution, use the default SQL store database options as described in the previous section.
To learn more about ACID and BASE compliance, see the AWS article What’s the difference between an ACID and a BASE database?.
The C3 Agentic AI Platform persists data for any of these solutions when you hibernate or stop an environment.
Requirement for specifying a data storage solution
You must have the C3.ClusterAdmin role to set up the configuration for a database that is not Cassandra, H2, or PostgreSQL.
If a user with the C3.ClusterAdmin role has already set up the configuration, any user can add the Ann.Db annotation to their Types to specify any of the data storage options.
Use the default KV store determined by your environment
Here's how to use the default KV store as the data store solution for an entity Type:
To configure a Type to use the default KV store solution, add the Ann.Db annotation and define the datastore field as kv. Here's an example Type KvTestType with the annotation:
@db(datastore="kv",
partitionKeyField='pk',
persistenceOrder="timestamp")
entity type KvTestType mixes Partitionable<string>The field datastore="kv" determines the data storage solution based on your environment. For SNEs, datastore="kv" defaults to the H2 database. For SEs and SNEs with a shared database, datastore="kv" defaults to the Cassandra database for KV store.
Specify Cassandra, H2, or PostgreSQL data storage
To configure an entity Type to use a Cassandra, H2, or PostgreSQL, add the Ann.Db annotation and define the datastore field as cassandra to use Cassandra as a database, or sqlkvstore to use H2 or PostgreSQL as a KV store. Your environment determines whether the Type uses H2 or PostgreSQL as a database, as defined by the table in the previous section, "KV Store solutions that you can specify".
Here is an example Type KvTestType with the cassandra annotation:
@db(datastore="cassandra",
partitionKeyField='pk',
persistenceOrder="timestamp")
entity type KvTestType mixes Partitionable<string>Specify the file system KV store
The file system KV store uses your application's default file system for data persistence instead of a database. The file system KV store persists data partitioned by Type name and partition key field. The file system KV stores data in your cloud provider's blob storage.
To learn more about how to access and work with the file system and file system mounts, see Work With File Systems.
To configure a Type to use the C3 AI file system storage, add the Ann.Db annotation and define the datastore field as fs. Here's an example Type User.Event.Login with the annotation:
@db(datastore="fs",
partitionKeyField='userId',
persistenceOrder='date')
entity type User.Event.Login mixes User.Event, Partitionable<string>, Configurable<User.Event.Login.Config> Other alternatives
C3 AI supports DynamoDB and Bigtable as alternative data storage options.
- To configure a Type to use DynamoDB or Bigtable, add the Ann.Db annotation to store data in a specified data store. Define the
datastorefield as your KV store alternative. Here's an example Type DynDB with the annotation:
@db(datastore="dyndb", partitionKeyField='pk', persistenceOrder='timestamp')
entity type MyMeasurement mixes Partitionable<string> {
pk: string
timestamp: datetime
}- Run the following command in the C3 AI Console to configure an alternative data storage connection in the C3 Agentic AI Platform:
config = DynamoDBConfig.make({name: "dyndb", credentials: {serverEndpoint: "<ServerEndpoint>", port: "<Port>", username: "<Username>", password: "<Password>"}})
config.setConfig("ConfigOverride.<level>")
config.setSecret("<level>")Replace <DynamoDBConfig> with the name of the configuration file, and replace <level> with ENV, APP, or CLUSTER. The default <level> is APP if not defined.
- Restart your environment, app, or cluster to complete the configuration.
<level>.restart()Replace <level> with Env, App, or Cluster.