Work With File Systems

To load data from files into the C3 Agentic AI Platform, the platform must have access to the file system where the files are stored. The platform comes with a configured local file system. The platform also has connectors to popular cloud file systems, such as Amazon S3, Azure Blob storage, and Google Cloud Storage. For each one, see Ingest Files from Amazon S3 Programmatically, Azure Blob Connector, and Google Cloud Storage Connector.

You can connect to file systems in two ways:

• Data Fusion in C3 AI Studio (see Data Fusion Overview)
• The FileSystem Type

The FileSystem Type presents remote object stores and network drives as standard file systems. With the FileSystem Type, you can view and set file system mounts and change the default file system. To access a remote file system, first connect it to the C3 Agentic AI Platform by following its connector page.

After you connect a file system, instantiate it with the following:

// Access the default file system
FileSystem.inst();

// Access the local file system
FileSystem.c3();

// Access the S3 file system
FileSystem.s3();

// Access the Azure Blob file system
FileSystem.azure();

// Access the Google Cloud Storage file system
FileSystem.gcs();

Copy link to this section

FileSystem mounts

To interact with the contents of a file system in a C3 AI Application, you need a file system mount. A mount points to a directory that contains the files. When you create an environment or application, the C3 Agentic AI Platform creates a set of file system mounts private to it. These mounts are subdirectories within the default bucket or object storage of the cluster, but other applications cannot view or modify them.

Copy link to this section

View mounts

You can view all mounts on an application with the following command:

// Checking the mounts of the default file system
FileSystem.inst().mounts();

The command returns the mounts associated with the default file system. The following example shows the returned mounts:

{
    "artifact": "gcs://c3--platform/artifacts/",
    "system": "gcs://c3--platform/av85/windturbine/system/",
    "attachment": "gcs://c3--platform/av85/windturbine/attachment/",
    "data-load": "gcs://c3--platform/av85/windturbine/dl/",
    "key-value": "gcs://c3--platform/av85/windturbine/kv/",
    "telemetry": "gcs://c3--platform/av85/windturbine/telemetry/",
    "datasets": "gcs://c3--platform/datasets/",
    "etl": "gcs://c3--platform/av85/windturbine/etl/",
    "/": "gcs://c3--platform/av85/windturbine/fs/"
}

In this result, two mounts are shared, artifact and datasets, so that applications can share files. The other mounts on an application always use the following format (note the <env_name> and <app_name> in the paths):

<cloud_provider>://<bucket_name>/<env_name>/<app_name>/<mount_name>

The platform can read and write to any child directory of a mounted file path. The configured credential must have the matching privilege on the cloud provider's side. To learn how to work with files, see Work With Files.

Copy link to this section

Default mount

The platform uses many of these file system mounts for its core interactions with the default file system (see the FileSystemMount Type for more information). The / mount is the root, or default, mount path for the connected file system. You can see the path by running the following:

// Checking the default mount of the default file system
FileSystem.inst().urlFromMount();

The command returns the default mount path, gcs://c3--platform/av85/windturbine/fs/ in the example result. Alternatively, you can see the default mount path by accessing the / element from the list of mounts:

// Get the `/` mount path from the list of all mounts
FileSystem.inst().mounts().get('/');

Copy link to this section

Set new mounts

You can also add mount paths to new file systems. A new mount helps when you have an existing remote file system whose contents you want to access from the C3 Agentic AI Platform. You can set a mount path at the application, environment, or cluster level.

The naming convention for mounts on the C3 Agentic AI Platform is kebab-case. Use the following code example to add a new file system mount path or update an existing file system mount path.

var mountName = "<my-mount-name>"; // e.g. 'my-new-mount'
var path = "<my_mount_path>" // e.g. 'gcs://my-bucket/my-new-mount-path/'
var overrideLevel = ConfigOverride.APP // change to ConfigOverride.ENV or ConfigOverride.CLUSTER with caution

// Create or update a mount path for the default file system 
FileSystem.inst().setMount(mountName, path, overrideLevel);
FileSystem.inst().mounts().get(mountName);

The last line returns the mount path that you just created. You can also see your new mount when you run FileSystem.inst().mounts().

Copy link to this section

Common issues when setting a mount

When you configure a new mount from within an application, you cannot set mounts that belong to other applications. You also cannot create mounts outside your application to access other files in the same bucket. For example, suppose your default mount is gcs://c3--platform/av85/windturbine/fs/. If you set a mount to gcs://c3--platform/c3/c3/ (to access files from the c3/c3 application), you get the following error:

// Run in av85/windturbine
FileSystem.inst().setMount('c3-mount', 'gcs://c3--platform/c3/c3/', ConfigOverride.APP);

// Fails with the following error
> Uncaught Error: Error invoking Java method GcsFileSystem#setMount: 
> Not permitted to add gcs://c3--platform/c3/c3/ url to mount c3-mount

To share files between applications, use the datasets mount.

Copy link to this section

Change the default file system

You can change the default file system to point to a different file system. Check the configured default file system with the following:

// Checking the scheme of the default file system
FileSystem.inst().scheme(); 

After connecting to the remote file system, a Cluster Administrator (ClusterAdmin) can change the default remote file system with the following commands:

// Make Amazon S3 the default file system
FileSystem.s3().makeDefault();

// Make an Azure Blob storage account the default file system
FileSystem.azure().makeDefault();

// Make Google Cloud Storage the default file system
FileSystem.gcs().makeDefault();

Copy link to this section

See also

• Work With Files

• Ingest Files from Amazon S3 Programmatically

• Integrate Data Programmatically

• Upload Files