C3 AI Documentation Home

File System Configuration Example — Amazon S3

Data Fusion supports a wide range of connectors for integrating data from file systems, databases, and streaming sources.

This topic provides a detailed configuration example for the Amazon S3 connector and demonstrates the connector setup workflow, bucket and mount configuration behavior, authentication handling, and validation process used in Data Fusion.

All connectors follow a consistent creation and configuration experience in Data Fusion, using the unified connector workflow and canvas-based representation.

The S3 connector enables Data Fusion to access data stored in Amazon S3 buckets and use it as a source for ingestion pipelines.

Connector Representation

Once created, the S3 connector:

  • Appears as a Source System (FileSourceSystem) node on the canvas
  • Serves as the entry point for file-based ingestion workflows
  • Can be connected to a Source Collection to define ingestion logic

Connection Configuration Model

S3 connections are configured using a guided setup workflow that dynamically adjusts based on the selected bucket configuration.

The connector workflow includes the following setup stages:

  • Connector — Select the connector type
  • Destination — Configure connector and bucket information
  • Auth — Configure authentication settings when required
  • Review — Validate and save the connector configuration

The number of setup stages varies depending on the selected bucket type and configuration method.

Connector Information

The connector configuration workflow includes metadata fields for the connector:

  • Name — Unique identifier used within the application
  • Description — Optional context for the connector

Bucket and Mount Configuration

In Data Fusion, reusable bucket connection configurations are managed as mounts. The Bucket Information section supports both platform-managed and user-created mounts.

Platform-Managed Mounts

Platform-managed mounts are system-provided bucket configurations managed by the platform.

For platform-managed mounts:

  • Authentication is managed by the platform
  • Credentials cannot be overridden
  • Users can select the mount directly from the existing bucket list

User-Created Mounts

User-created mounts are bucket configurations created by users through the Add Bucket workflow.

After creation, user-created mounts appear in the Use Existing Bucket list and can be reused in future connector configurations.

For user-created mounts:

  • Users configure authentication during bucket creation
  • Credentials can later be overridden if needed

Configure Existing Buckets

When using an existing bucket:

  • Select Use Existing Bucket
  • Choose a configured mount from the Bucket list
  • Existing bucket configurations reuse the authentication settings associated with the selected mount

Depending on the selected mount type:

  • Platform-managed mounts skip the Auth step entirely
  • User-created mounts may allow credential overrides

Add Bucket Workflow

Use Add Bucket to create a new reusable user-created mount configuration.

When creating a new bucket:

  • Enter the bucket path and select the AWS region
  • The workflow includes an Auth step
  • Authentication details are configured and stored with the mount
  • The Setup Progress indicator dynamically updates to include the Auth stage
  • After creation, the mount becomes available under Use Existing Bucket

Auto-populate Configuration

The connector workflow includes an Auto-populate option that allows you to populate Destination, Auth, and Advanced configuration fields at once.

Selecting Auto-populate opens the Auto-Map Fields panel, which guides you through the following steps:

Provide Input

Provide connection configuration details using one of the following input types:

  • JSON — Paste a JSON configuration directly or upload a JSON file
  • JDBC URL — Provide a JDBC URL to extract connection details

Map Extracted Keys to Connector Fields

After the configuration is processed, Data Fusion suggests mappings between the extracted configuration keys and connector fields.

Review the suggested mappings and adjust them as needed before applying the configuration.

Note:

Only configuration keys are used for field mapping. Actual configuration values remain private and are not exposed during the mapping process.

Authentication Behavior

Authentication fields vary depending on the selected mount type and authentication method.

IAM-based methods (for example, IAMRole):

  • Do not require manual access key or secret key configuration
  • Resolve credentials from the execution environment

Credential-based methods may require:

  • Access key
  • Secret key
  • Additional authentication parameters

Validation and Persistence

The Review step displays the finalized connector configuration before validation.

Connection validation is performed using Save & Test.

A successful validation:

  • Confirms the connector configuration
  • Saves the connector
  • Makes the connector available on the Data Fusion canvas

If validation fails, verify the bucket configuration, credentials, and network access settings before retrying the connection test.

You can then select Save & Close to return to the canvas.

Connector Management

After creation, the S3 connector can be managed directly from the Data Fusion canvas.

The Source System node menu supports actions such as:

  • Unlinking the connector from the pipeline
  • Deleting the connector

Unlinking a connector removes the connector relationship from the current pipeline but does not delete the connector configuration from the environment.

Add an S3 Connector in Data Fusion

Configure an Amazon S3 connector in Data Fusion by selecting the S3 connector, configuring the destination and bucket settings, validating the connection, and saving the connector to the Data Fusion canvas.

Prerequisites

Before starting, ensure you have the following:

  • A C3 environment running Version 8.11 or later
  • A running C3 application
  • Access to an Amazon S3 bucket
  • Required authentication credentials or IAM permissions
  • CSV-formatted files for structured ingestion workflows

Steps

  1. Open your application in C3 AI Studio.
  2. Navigate to Data Fusion.
  3. On the canvas, select Add Data Source.
  4. In the Configure Connector window: a. Select S3 under File systems
  5. In the connector configuration workflow, configure the following:

Connector Information

  • Name — Enter a unique connector name
  • Description — (Optional) Enter connector details

Bucket Information

Choose one of the following options:

Use Existing Bucket

  1. Select Use Existing Bucket.
  2. From the Bucket list, select either: a. A platform-managed mount b. A previously configured user-created mount
  3. Select the AWS region from the Region list.
  4. Continue to the next step.

Note:

  • Platform-managed mounts use platform-managed authentication and do not allow credential overrides.
  • User-created mounts reuse previously configured authentication settings and may allow credential overrides.

Add Bucket

  1. Select Add Bucket.
  2. Enter the bucket configuration details.
  3. Select the AWS region from the Region list.
  4. Continue to the Auth step.
  5. Configure the required authentication settings for the new mount.
  6. Complete the remaining configuration steps.

Note:

After the connector is created, the new mount becomes available under Use Existing Bucket for future connector configurations.

Review and Validate

  1. In the Review step: a. Review the connector configuration details b. Verify the bucket and authentication settings
  2. Select Save & Test to validate the connection.
  3. After validation succeeds: a. Select Save & Close

Result

  • The S3 connector is created
  • A Source System (FileSourceSystem) node appears on the Data Fusion canvas
  • The connector is available for pipeline configuration
  • The connector can be linked to a Source Collection for ingestion workflows

Next Steps

  • Connect the S3 connector to a Source Collection
  • Configure file paths and ingestion settings
  • Add downstream processing components such as transformations or processing nodes
  • Execute and monitor the ingestion pipeline

See also

Was this page helpful?