C3 AI Documentation Home

Work With Files

When working with large datasets stored in cloud storage, you might want to access and view all files across various directories to facilitate processing, importing, or analysis. The C3 Agentic AI Platform offers straightforward APIs for performing all read and write operations on files. This topic provides sample code for both the JavaScript and Python SDKs.

This topic assumes familiarity with the topic Work with File Systems. If you are not familiar with file system mounts, it is advisable to review that material prior to proceeding. The topic is divided into the following sections:

  • List files and directories
  • Make a File instance
  • Read files
  • Write to files
  • Copy and move files
  • Delete files
  • Upload and download files in Python

List files and directories

Utilize the FileSystem#listFiles API in the C3 Agentic AI Platform to view files within a file system.

JavaScript
// JAVASCRIPT
// Retrieve files from the default mount on the default file system
var result = FileSystem.listFiles();
// View the results
c3Grid(result.files)
Python
# PYTHON
# Retrieve files from the default mount on the default file system
result = c3.FileSystem.listFiles()
# View the results
[r for r in result.files]

If you do not pass any parameters into FileSystem.listFiles(), C3 uses the 'default', or '/' mount. Passing an empty string ("") is equivalent to not passing any parameters at all. To view files on different mounts or to access other file paths, pass a URL as the first parameter:

JavaScript
// JAVASCRIPT
// Retrieves the data-load mount for your app
var dataLoadMount = FileSystem.mounts().get('data-load');
// Retrieves files at the URL above
var result = FileSystem.listFiles(dataLoadMount, -1);
// View the results
c3Grid(result.files)
Python
# PYTHON
# Retrieves the data-load mount for your app
dataLoadMount = c3.FileSystem.mounts()['data-load']
# Retrieves files at the URL above
result = c3.FileSystem.listFiles(dataLoadMount, -1)
# View the results
[r for r in result.files]

The -1 in the second parameter controls the number of files to return (-1 is no limit). By default, only 100 files are returned.

List directories

You can also list subdirectories of a file location, which is particularly useful when dealing with millions of files but only a few dozen unique directories within a mount path.

JavaScript
// JAVASCRIPT
// Ex, retrieves the data-load mount for your app
var dataLoadMount = FileSystem.mounts().get('data-load');
// Creates a stream of all directories on the data-load mount 
var stream = FileSystem.listDirsStream(dataLoadMount, -1);
// USE WITH CAUTION - collects the entire stream into memory
var directories = stream.collect();
Python
# PYTHON
# Ex, retrieves the data-load mount for your app
dataLoadMount = c3.FileSystem.mounts()['data-load']
# Creates a stream of all directories on the data-load mount 
stream = c3.FileSystem.listDirsStream(dataLoadMount, -1)
# USE WITH CAUTION - collects the entire stream into memory
directories = stream.collect()

List all files on all mounts

When you inspect the output of FileSystem.mounts(), you notice an 'fs/' suffix on the '/' mount. This prevents you from seeing files on all mounts in your application by default. While you can list files on all mounts in a loop and concatenate the results, an easier method to see all files is to set a mount one folder higher and then use that mount in FileSystem#listFiles.

JavaScript
// JAVASCRIPT
// Get the default mount URL (equivalent to FileSystem.mounts().get('/'))
var defaultMount = FileSystem.rootUrl();
// Remove the 'fs/' suffix from the end
var trueRoot = defaultMount.substring(0, defaultMount.length - 3);
// Set the new URL as a file system mount
FileSystem.setMount('true-root', trueRoot);
// List all files at the new URL
var result = FileSystem.listFiles(trueRoot, 1000); // Second parameter limit (default is 100, but -1 is no limit)
Python
# PYTHON
# Get the default mount URL (equivalent to FileSystem.mounts().get('/'))
defaultMount = c3.FileSystem.rootUrl()
# Remove the 'fs/' suffix from the end
trueRoot = defaultMount[:-3]
# Set the new URL as a file system mount
c3.FileSystem.setMount('true-root', trueRoot)
# List all files at the new URL
result = c3.FileSystem.listFiles(trueRoot, 1000) # Second parameter limit (default is 100, but -1 is no limit)

Make a File instance

FileSystem#listFiles is a useful API to view and retrieve files on the attached file system. You can also make a File instance in memory with FileSystem#makeFile. When making a File, the C3 Agentic AI Platform does not immediately check for the existence of the file, but the instance remains usable in methods, scripts, or commands.

The following example shows a file my-file.txt that is located in the default mount. If you only pass in a file name (not a fully formed file URL), the C3 Agentic AI Platform uses the default mount as the base URL for the file.

If the file my-file.txt exists, you can check its metadata and read the existing content. If the file does not exist, no associated metadata appears, but you can interact with the file and write content to the path.

JavaScript
// JAVASCRIPT
// Ex 1: Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);

// Ex 2: Make a file at the data-load mount
var mount = FileSystem.mounts().get('data-load');
var fileUrl = mount + fileName;
var file = FileSystem.makeFile(fileUrl);
Python
# PYTHON
# Ex 1: Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)

# Ex 2: Make a file at the data-load mount
mount = c3.FileSystem.mounts()['data-load']
fileUrl = mount + fileName
file = c3.FileSystem.makeFile(fileUrl)

Read files

After you make a File instance, use File#readString to read its content.

JavaScript
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);

// Read the first 100 characters from the file
console.log(file.readString(0, 100));
Python
# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)

# Read the first 100 characters from the file
print(file.readString(0, 100))

When using readString() on large files, pass in the two parameters above (start, end character) to limit the returned content. If you do not pass in any parameters, the entire file is read into memory.

For non-text files, utilize File#read to retrieve the binary content.

JavaScript
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);

// Read all bytes from the file
file.read();
Python
# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)

# Read all bytes from the file
file.read()

Check file metadata

With a File instance, it is also possible to retrieve and view the file metadata.

JavaScript
// JAVASCRIPT
// Open a file at the default mount URL and view metadata (if it exists)
var fileName = 'my-file.txt';
var file = FileSystem.openFile(fileName);

// Check if File exists
file.exists();
Python
# PYTHON
# Open a file at the default mount URL and view metadata (if it exists)
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)

# Check if File exists
file.exists()

Write to files

In the above example, it is likely that my-file.txt does not exist on the file system. Content is written to a file URL using File#writeString. In the below example, the text "Hello File#writeString!" is written to the file.

JavaScript
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);

// Write to the file location
file.writeString('Hello File#writeString!');

// Optional - this should return true now
file.exists();
// Optional - read the new content
console.log(file.readString());
Python
# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)

# Write to the file location
file.writeString('Hello File#writeString!')

# Optional - this should return true now
file.exists()
# Optional - read the new content
print(file.readString())

File#writeString overwrites any existing content at the file location - it does not append to the existing content. To append content to an existing file, read the content into memory, append your text, and then write the new content back to the same location. An example is shown below:

JavaScript
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);
// Write to the file location
file.writeString('Hello File#writeString!');
// Read the content into memory
var content = file.readString();
// Append a second line of text to the content
content += '\nLine 2 of my file';
// Write the new content back to the file location
file.writeString(content);

// Optional: View the new content
console.log(file.readString());
Python
# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)
# Write to the file location
file.writeString('Hello File#writeString!')
# Read the content into memory
content = file.readString()
# Append a second line of text to the content
content += '\nLine 2 of my file'
# Write the new content back to the file location
file.writeString(content)

# Optional: View the new content
print(file.readString())

Copy and move files

The C3 Agentic AI Platform provides straightforward APIs for copying or moving files to new directories.

Copy a file to a new location

There are two methods to copy a File to a new location, depending on if you are copying one file, or if you are copying multiple files at once.

Copy one file: To copy one file, use File#copy, where you pass in the full URL (including file name on the destination URL) to copy a file:

JavaScript
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);
// Write to the file location
file.writeString('Hello File#copy!');

// Copy the file to the data-load mount in this example
var destinationUrl = FileSystem.mounts().get('data-load');

// IMPORTANT: Provide the fileName in the copy() command
var newFile = file.copy(destinationUrl + fileName);
Python
# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)
# Write to the file location
file.writeString('Hello File#copy!')

# Copy the file to the data-load mount in this example
destinationUrl = c3.FileSystem.mounts()['data-load']

# IMPORTANT: Provide the fileName in the copy() command
newFile = file.copy(destinationUrl + fileName)

The File#copy command returns an instance of the newly created file.

Copy multiple files: To copy multiple files at once, utilize FileSystem#copyFiles. With this command, you do not pass in fully qualified file names - you pass in the directories that you are copying files between. The C3 Agentic AI Platform looks in the source directory for all files and copies them to the new directory.

JavaScript
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);
// Write to the file location
file.writeString('Hello FileSystem#copyFiles!');

// Copy the files to the data-load mount in this example
var destinationUrl = FileSystem.mounts().get('data-load');
var defaultMount = FileSystem.rootUrl();

// Only passing in directories, NOT full file names
FileSystem.copyFiles(defaultMount, destinationUrl);
Python
# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)
# Write to the file location
file.writeString('Hello FileSystem#copyFiles!')

# Copy the files to the data-load mount in this example
destinationUrl = c3.FileSystem.mounts()['data-load']
defaultMount = c3.FileSystem.rootUrl()

# Only passing in directories, NOT full file names
c3.FileSystem.copyFiles(defaultMount, destinationUrl)

This command returns the number of files that were copied to the new directory.

Move files

To move files between directories, the C3 Agentic AI Platform provides File#move and FileSystem#moveFiles commands, which utilize the same parameters as File#copy and FileSystem#copyFiles.

Move one file: To move one file, use File#move, where you pass in the full URL (including file name on the destination URL) to move a file:

JavaScript
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);
// Write to the file location
file.writeString('Hello File#move!');

// Move the file to the data-load mount in this example
var destinationUrl = FileSystem.mounts().get('data-load');

// IMPORTANT: Provide the fileName in the move() command
var newFile = file.move(destinationUrl + fileName);
Python
# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)
# Write to the file location
file.writeString('Hello File#move!')

# Move the file to the data-load mount in this example
destinationUrl = c3.FileSystem.mounts()['data-load']

# IMPORTANT: Provide the fileName in the move() command
newFile = file.move(destinationUrl + fileName)

The File#move command returns an instance of the newly created file.

Move multiple files: To move multiple files at once, utilize FileSystem#moveFiles. With this command, you do not pass in fully qualified file names - you pass in the directories that you wish to move files between. The C3 Agentic AI Platform looks in the source directory for all files and moves them to the new directory.

JavaScript
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);
// Write to the file location
file.writeString('Hello FileSystem#moveFiles!');

// Move the files to the data-load mount in this example
var destinationUrl = FileSystem.mounts().get('data-load');
var defaultMount = FileSystem.rootUrl();

// Only passing in directories, NOT full file names
FileSystem.moveFiles(defaultMount, destinationUrl);
Python
# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)
# Write to the file location
file.writeString('Hello FileSystem#moveFiles!')

# Move the files to the data-load mount in this example
destinationUrl = c3.FileSystem.mounts()['data-load']
defaultMount = c3.FileSystem.rootUrl()

# Only passing in directories, NOT full file names
c3.FileSystem.moveFiles(defaultMount, destinationUrl)

This command returns the number of files that were moved to the new directory.

Delete files

Delete files with the File#delete command.

JavaScript
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);
// Write to the file location
file.writeString('Hello File#delete!');

// Delete the File
file.delete();

// Returns null
file.readString();
// Returns false
file.exists();
Python
# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)
# Write to the file location
file.writeString('Hello File#delete!')

# Delete the File
file.delete()

# Returns null
file.readString()
# Returns false
file.exists()

Upload and download files in Python

In the C3 Agentic AI Platform, you can upload and download local files to and from a remote file system. This is extremely useful to bring remote files (e.g. from AWS S3) into your local container when working in Jupyter, and it is also useful when leveraging files (e.g. PDFs, images) inside an MlPipeline. Downloading a remote file to the local file system enables lower latency and more native code options for working with files.

Local files that are created using native Python code are not tracked or managed by the C3 Agentic AI Platform. Manage and delete local files after you are finished with them to avoid excessive usage of the local file system on the underlying nodes.

Download a remote file from the remote file system

The recommended way to download is to generate a presigned GET URL with FileSystem#generatePresignedUrl, then stream it into a local file using standard Python libraries.

Python
import requests

# Make a file at the default mount URL
fileName = "my-file.txt"
remote_file = c3.FileSystem.makeFile(fileName)

# Write some content remotely
remote_file.writeString("Hello world!")

# Generate a presigned GET URL
get_url = c3.FileSystem.generatePresignedUrl(remote_file.url, method="GET", expiration="1h")

# Choose a local file path and stream download
local_file_path = "./my-file.txt"
with requests.get(get_url, stream=True) as r:
    r.raise_for_status()
    with open(local_file_path, "wb") as f:
        for chunk in r.iter_content(chunk_size=1024*1024):
            if chunk:
                f.write(chunk)

Upload a local file to the remote file system

Use FileSystem#generatePresignedUrl with method "PUT" to stream a local file to the remote system.

Python
import requests

# Make a local file and write content
local_file_path = "./my-file.txt"
with open(local_file_path, "w") as f:
    f.write("Hello world!")

# Make a file at the default mount URL
fileName = "my-file.txt"
remote_file = c3.FileSystem.makeFile(fileName)

# Generate a presigned PUT URL for upload
put_url = c3.FileSystem.generatePresignedUrl(remote_file.url, method="PUT", expiration="1h")

# Read local file and upload
with open(local_file_path, "rb") as f:
    data = f.read()
    headers = {
        "Content-Type": "text/plain",           # required for PUT
        "Content-Length": str(len(data)),       # required for PUT
        # For Azure Blob also add: "x-ms-blob-type": "BlockBlob"
    }
    resp = requests.put(put_url, data=data, headers=headers)
    resp.raise_for_status()

# Verify by reading remote file back
print(remote_file.readString(0, 100))

Handle file operations in an environment without writing to a file

In some scenarios, you may not want to create a local file before uploading content to the remote file system (e.g. when pickling a model). In this case, utilize native Python libraries such as tempfile to assist with operations that would normally require a local file to be created. A key benefit is that you do not need to worry about managing the local file after it is uploaded to the remote file system.

In the following example, a local_dict object is serialized as a YAML string before being uploaded the remote file system as local_dict.yaml.

Python
from tempfile import NamedTemporaryFile
import os
import yaml

# Create a local python object
local_dict = {'id': '123', 'data': 42}

with NamedTemporaryFile() as temp_file:
  # Create the temp file and write to it
  dumped_yaml = yaml.dump(local_dict)
  temp_file.write(bytes(dumped_yaml,encoding='utf8'))
  temp_file.seek(0)
  # Form the remote file URL
  dest_url = c3.FileSystem.rootUrl() + 'local_dict.yaml'
  # Write the content to the remote file location
  c3.FileSystem.uploadFile(srcFile=temp_file, dstPath=dest_url)

You can use any library that supports writing to a file to upload something to the FileSystem.

Was this page helpful?