C3 AI Documentation Home

Amazon S3 Connector

The C3 Agentic AI Platform has a built-in connector for integrating with Amazon S3 file systems. To integrate with a new Amazon S3 bucket, you must:

  1. If a bucket does not already exist, create a bucket in AWS with the appropriate bucket access policies.
  2. Create a mount path for the S3 bucket in the C3 Agentic AI Platform, if it does not already exist.
  3. Set the required credentials to access the contents of the S3 bucket.
  4. Validate the connection.

Create an Amazon S3 bucket

Before connecting to an S3 bucket, create or use an existing bucket in AWS. For more information on creating an Amazon S3 bucket, see Creating a bucket in the AWS documentation.

To access the bucket from the C3 Agentic AI Platform, you must also create a role with the required permissions and apply any required policies on the S3 bucket. For more information, see the Amazon S3 documentation on Access Control.

Minimum permissions needed for read-only access

To grant read-only access in Amazon S3, the minimum required role is AmazonS3ReadOnlyAccess. This permission should be included in the IAM role or bucket policy.

Platform configurations

After you create a bucket and apply the appropriate access control policies, apply the following configurations on the C3 Agentic AI Platform.

Enable the file system

Run the following command to enable the S3 remote file system:

JavaScript
FileSystem.s3().enable();

For more information on enabling remote file systems on the C3 Agentic AI Platform, see Work With File Systems.

Create the file system mount

Use the following to add a new file system mount path for S3:

JavaScript
var mountName = "<mount_name>";
var bucketName = "<s3_bucket_name>";

FileSystem.s3().setMount(mountName, "s3://" + bucketName, ConfigOverride.APP);

Set credentials to access the Amazon S3 bucket

When setting the credentials to access the contents of your S3 bucket, the C3 AI Platform supports several options depending on the information security requirements of your organization. The recommended option is to generate an access key and secret key on behalf of an IAM user that has the appropriate level of access on the AWS side and set the credential as follows:

JavaScript
var bucketName = "<s3_bucket_name>";
var accessKey = "<access_key>";
var secretKey = "<secret key>";
var region = "<s3_bucket_region>";

var credentials = Aws.inst()
  .defaultCredentials()
  .withRegion(region)
  .withAccessKey(accessKey)
  .withSecretKey(secretKey);
AwsS3Bucket.setCredentialsForResourceName(
  bucketName,
  credentials,
  ConfigOverride.APP
);

You can copy existing credentials to a new S3 bucket using the following code:

JavaScript
var otherBucketName = "<name_of_the_existing_bucket>";
var bucketName = "<s3_bucket_name>";
var region = "<s3_bucket_region>";

AwsCredentials.make(
  AwsS3Bucket.forResourceName(otherBucketName).credentials
).withRegion(region);
AwsS3Bucket.setCredentialsForResourceName(
  bucketName,
  credentials,
  ConfigOverride.APP
);

To minimize security risk vectors, you should consider using a least privileges approach when establishing IAM roles and setting credentials, especially in production environments.

Connect to an external S3 bucket in a running cluster

If the bucket uses a custom endpoint, you must explicitly set the nonstandard endpoint in the AwsCredentials object as shown.

JavaScript
var bucketName = "<s3_bucket_name>";
credentials = AwsCredentials.make({
    "type": "AwsCredentials",
    "secretKey": "============================",
    "accessKey": "=============",
    "region":"us-east-1",
    "endpoint": "<nonstandard-aws-endpoint>"
  });
JavaScript
AwsS3Bucket.setCredentialsForResourceName(
bucketName,
credentials,
ConfigOverride.CLUSTER
);

Select the appropriate level for the configuration override to ensure that credentials and other settings are applied correctly based on the desired scope of influence. For instance, if the intention is to limit the credential settings to a specific application, using APP would be more appropriate than CLUSTER.

Validate the connection

After you apply the required configurations, validate that the S3 bucket has been correctly integrated by listing the files in the bucket:

JavaScript
var mountName = "<mount_name>";

FileSystem.s3().listFiles(FileSystem.s3().urlFromMount(mountName));

This request succeeds if the configuration has been correctly applied, even if there are no files in the bucket.

Clear credentials

To clear credentials for an S3 bucket and delete the associated mount path, run the following:

JavaScript
var bucketName = "<s3_bucket_name>";
var mountName = "<mount_name>";

AwsS3Bucket.forResourceName(bucketName).clearConfigAndSecretAllOverrides();
FileSystem.s3().removeMount(mountName, ConfigOverride.APP);

See also

Was this page helpful?