Work With Files
When you work with large datasets in cloud storage, you might want to access and view all files across directories to process, import, or analyze them. The C3 Agentic AI Platform offers APIs for all read and write operations on files. This topic provides sample code for both the JavaScript and Python SDKs.
This topic assumes familiarity with the topic Work With File Systems. If you are not familiar with file system mounts, review that topic first. The sample code shows how to:
- List files and directories
- Make a File instance
- Read files
- Write to files
- Copy and move files
- Delete files
- Upload and download files in Python
List files and directories
Use the FileSystem#listFiles API in the C3 Agentic AI Platform to view files within a file system.
// JAVASCRIPT
// Retrieve files from the default mount on the default file system
var result = FileSystem.listFiles();
// View the results
c3Grid(result.files)# PYTHON
# Retrieve files from the default mount on the default file system
result = c3.FileSystem.listFiles()
# View the results
[r for r in result.files]If you do not pass any parameters into FileSystem.listFiles(), the platform uses the default (/) mount. Passing an empty string ("") is equivalent to not passing any parameters at all. To view files on different mounts or to access other file paths, pass a URL as the first parameter:
// JAVASCRIPT
// Retrieves the data-load mount for your app
var dataLoadMount = FileSystem.mounts().get('data-load');
// Retrieves files at the URL above
var result = FileSystem.listFiles(dataLoadMount, -1);
// View the results
c3Grid(result.files)# PYTHON
# Retrieves the data-load mount for your app
dataLoadMount = c3.FileSystem.mounts()['data-load']
# Retrieves files at the URL above
result = c3.FileSystem.listFiles(dataLoadMount, -1)
# View the results
[r for r in result.files]The -1 in the second parameter controls the number of files to return (-1 is no limit). By default, the method returns at most 100 files. When more files exist, the result includes a nextMarker. Pass that value as the third parameter to list the next files.
List directories
You can also list subdirectories of a file location. This helps when a mount path holds millions of files but only a few dozen unique directories.
// JAVASCRIPT
// Ex, retrieves the data-load mount for your app
var dataLoadMount = FileSystem.mounts().get('data-load');
// Creates a stream of all directories on the data-load mount
var stream = FileSystem.listDirsStream(dataLoadMount, -1);
// USE WITH CAUTION - collects the entire stream into memory
var directories = stream.collect();# PYTHON
# Ex, retrieves the data-load mount for your app
dataLoadMount = c3.FileSystem.mounts()['data-load']
# Creates a stream of all directories on the data-load mount
stream = c3.FileSystem.listDirsStream(dataLoadMount, -1)
# USE WITH CAUTION - collects the entire stream into memory
directories = stream.collect()List all files on all mounts
When you inspect the output of FileSystem.mounts(), you notice an 'fs/' suffix on the '/' mount. This suffix prevents you from seeing files on all mounts in your application by default. You can list files on all mounts in a loop and concatenate the results. An easier method is to set a mount one folder higher and then use that mount in FileSystem#listFiles.
// JAVASCRIPT
// Get the default mount URL (equivalent to FileSystem.mounts().get('/'))
var defaultMount = FileSystem.rootUrl();
// Remove the 'fs/' suffix from the end
var trueRoot = defaultMount.substring(0, defaultMount.length - 3);
// Set the new URL as a file system mount
FileSystem.setMount('true-root', trueRoot);
// List all files at the new URL
var result = FileSystem.listFiles(trueRoot, 1000); // Second parameter limit (default is 100, but -1 is no limit)# PYTHON
# Get the default mount URL (equivalent to FileSystem.mounts().get('/'))
defaultMount = c3.FileSystem.rootUrl()
# Remove the 'fs/' suffix from the end
trueRoot = defaultMount[:-3]
# Set the new URL as a file system mount
c3.FileSystem.setMount('true-root', trueRoot)
# List all files at the new URL
result = c3.FileSystem.listFiles(trueRoot, 1000) # Second parameter limit (default is 100, but -1 is no limit)Make a File instance
FileSystem#listFiles views and retrieves files on the attached file system. You can also make a File instance in memory with FileSystem#makeFile. When you make a File, the C3 Agentic AI Platform does not immediately check that the file exists. The instance remains usable in methods, scripts, or commands.
The following example shows a file my-file.txt in the default mount. If you pass in only a file name instead of a full file URL, the platform uses the default mount as the base URL.
If the file my-file.txt exists, you can check its metadata and read the existing content. If the file does not exist, no associated metadata appears, but you can interact with the file and write content to the path.
// JAVASCRIPT
// Ex 1: Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);
// Ex 2: Make a file at the data-load mount
var mount = FileSystem.mounts().get('data-load');
var fileUrl = mount + fileName;
var file = FileSystem.makeFile(fileUrl);# PYTHON
# Ex 1: Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)
# Ex 2: Make a file at the data-load mount
mount = c3.FileSystem.mounts()['data-load']
fileUrl = mount + fileName
file = c3.FileSystem.makeFile(fileUrl)Read files
After you make a File instance, use File#readString to read its content.
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);
// Read the first 100 characters from the file
console.log(file.readString(0, 100));# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)
# Read the first 100 characters from the file
print(file.readString(0, 100))When you use readString() on large files, pass in the two parameters (start and end character) to limit the returned content. If you do not pass in any parameters, the method reads the entire file into memory.
For non-text files, use File#read to retrieve the binary content.
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);
// Read all bytes from the file
file.read();# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)
# Read all bytes from the file
file.read()Check file metadata
With a File instance, you can also retrieve and view the file metadata.
// JAVASCRIPT
// Make a file at the default mount URL and view metadata (if it exists)
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);
// Check if File exists
file.exists();
// View the metadata, such as contentLength and lastModified
file.readMetadata();# PYTHON
# Make a file at the default mount URL and view metadata (if it exists)
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)
# Check if File exists
file.exists()
# View the metadata, such as contentLength and lastModified
file.readMetadata()Write to files
In the previous example, my-file.txt likely does not exist on the file system. Use File#writeString to write content to a file URL. The following example writes the text "Hello File#writeString!" to the file.
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);
// Write to the file location
file.writeString('Hello File#writeString!');
// Optional - this should return true now
file.exists();
// Optional - read the new content
console.log(file.readString());# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)
# Write to the file location
file.writeString('Hello File#writeString!')
# Optional - this should return true now
file.exists()
# Optional - read the new content
print(file.readString())File#writeString overwrites any existing content at the file location. It does not append to the existing content. To append content to an existing file, read the content into memory, append your text, and then write the new content back to the same location. The following example shows how.
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);
// Write to the file location
file.writeString('Hello File#writeString!');
// Read the content into memory
var content = file.readString();
// Append a second line of text to the content
content += '\nLine 2 of my file';
// Write the new content back to the file location
file.writeString(content);
// Optional: View the new content
console.log(file.readString());# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)
# Write to the file location
file.writeString('Hello File#writeString!')
# Read the content into memory
content = file.readString()
# Append a second line of text to the content
content += '\nLine 2 of my file'
# Write the new content back to the file location
file.writeString(content)
# Optional: View the new content
print(file.readString())Copy and move files
The C3 Agentic AI Platform provides APIs for copying or moving files to new directories.
Copy a file to a new location
There are two methods to copy a File to a new location, depending on whether you copy one file or multiple files at once.
Copy one file: To copy one file, use File#copy and pass in the full destination URL, including the file name:
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);
// Write to the file location
file.writeString('Hello File#copy!');
// Copy the file to the data-load mount in this example
var destinationUrl = FileSystem.mounts().get('data-load');
// IMPORTANT: Provide the fileName in the copy() command
var newFile = file.copy(destinationUrl + fileName);# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)
# Write to the file location
file.writeString('Hello File#copy!')
# Copy the file to the data-load mount in this example
destinationUrl = c3.FileSystem.mounts()['data-load']
# IMPORTANT: Provide the fileName in the copy() command
newFile = file.copy(destinationUrl + fileName)The File#copy command returns an instance of the newly created file.
Copy multiple files: To copy multiple files at once, use FileSystem#copyFiles. With this command, you do not pass in fully qualified file names. You pass in the directories that you are copying files between. The C3 Agentic AI Platform looks in the source directory for all files and copies them to the new directory.
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);
// Write to the file location
file.writeString('Hello FileSystem#copyFiles!');
// Copy the files to the data-load mount in this example
var destinationUrl = FileSystem.mounts().get('data-load');
var defaultMount = FileSystem.rootUrl();
// Only passing in directories, NOT full file names
FileSystem.copyFiles(defaultMount, destinationUrl);# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)
# Write to the file location
file.writeString('Hello FileSystem#copyFiles!')
# Copy the files to the data-load mount in this example
destinationUrl = c3.FileSystem.mounts()['data-load']
defaultMount = c3.FileSystem.rootUrl()
# Only passing in directories, NOT full file names
c3.FileSystem.copyFiles(defaultMount, destinationUrl)This command returns the number of files that were copied to the new directory.
Move files
To move files between directories, the C3 Agentic AI Platform provides File#move and FileSystem#moveFiles commands, which take the same parameters as File#copy and FileSystem#copyFiles.
Move one file: To move one file, use File#move and pass in the full destination URL, including the file name:
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);
// Write to the file location
file.writeString('Hello File#move!');
// Move the file to the data-load mount in this example
var destinationUrl = FileSystem.mounts().get('data-load');
// IMPORTANT: Provide the fileName in the move() command
var newFile = file.move(destinationUrl + fileName);# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)
# Write to the file location
file.writeString('Hello File#move!')
# Move the file to the data-load mount in this example
destinationUrl = c3.FileSystem.mounts()['data-load']
# IMPORTANT: Provide the fileName in the move() command
newFile = file.move(destinationUrl + fileName)The File#move command returns an instance of the newly created file.
Move multiple files: To move multiple files at once, use FileSystem#moveFiles. With this command, you do not pass in fully qualified file names. You pass in the directories that you want to move files between. The C3 Agentic AI Platform looks in the source directory for all files and moves them to the new directory.
As written, the following example moves every file on the default mount, not only my-file.txt, so pass a narrower source path such as a folder.
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);
// Write to the file location
file.writeString('Hello FileSystem#moveFiles!');
// Move the files to the data-load mount in this example
var destinationUrl = FileSystem.mounts().get('data-load');
var defaultMount = FileSystem.rootUrl();
// Only passing in directories, NOT full file names
FileSystem.moveFiles(defaultMount, destinationUrl);# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)
# Write to the file location
file.writeString('Hello FileSystem#moveFiles!')
# Move the files to the data-load mount in this example
destinationUrl = c3.FileSystem.mounts()['data-load']
defaultMount = c3.FileSystem.rootUrl()
# Only passing in directories, NOT full file names
c3.FileSystem.moveFiles(defaultMount, destinationUrl)This command returns the number of files that were moved to the new directory.
Delete files
Delete files with the File#delete command.
// JAVASCRIPT
// Make a file at the default mount URL
var fileName = 'my-file.txt';
var file = FileSystem.makeFile(fileName);
// Write to the file location
file.writeString('Hello File#delete!');
// Delete the File
file.delete();
// Returns null
file.readString();
// Returns false
file.exists();# PYTHON
# Make a file at the default mount URL
fileName = 'my-file.txt'
file = c3.FileSystem.makeFile(fileName)
# Write to the file location
file.writeString('Hello File#delete!')
# Delete the File
file.delete()
# Returns null
file.readString()
# Returns false
file.exists()Upload and download files in Python
In the C3 Agentic AI Platform, you can upload and download local files to and from a remote file system. Use this to bring remote files, for example from Amazon S3, into your local container when you work in Jupyter. You can also use files, such as PDFs or images, inside an MlPipeline. Downloading a remote file to the local file system enables lower latency and more native code options for working with files.
The C3 Agentic AI Platform does not track or manage local files that you create with native Python code. Manage and delete local files after you finish with them to avoid excessive usage of the local file system on the underlying nodes.
Download a remote file from the remote file system
The recommended way to download is to generate a presigned GET URL with FileSystem#generatePresignedUrl, then stream it into a local file using standard Python libraries.
import requests
# Make a file at the default mount URL
fileName = "my-file.txt"
remote_file = c3.FileSystem.makeFile(fileName)
# Write some content remotely
remote_file.writeString("Hello world!")
# Generate a presigned GET URL
get_url = c3.FileSystem.generatePresignedUrl(remote_file.url, method="GET", expiration="1h")
# Choose a local file path and stream download
local_file_path = "./my-file.txt"
with requests.get(get_url, stream=True) as r:
r.raise_for_status()
with open(local_file_path, "wb") as f:
for chunk in r.iter_content(chunk_size=1024*1024):
if chunk:
f.write(chunk)Upload a local file to the remote file system
Use FileSystem#generatePresignedUrl with method "PUT" to stream a local file to the remote system.
import requests
# Make a local file and write content
local_file_path = "./my-file.txt"
with open(local_file_path, "w") as f:
f.write("Hello world!")
# Make a file at the default mount URL
fileName = "my-file.txt"
remote_file = c3.FileSystem.makeFile(fileName)
# Generate a presigned PUT URL for upload
put_url = c3.FileSystem.generatePresignedUrl(remote_file.url, method="PUT", expiration="1h")
# Read local file and upload
with open(local_file_path, "rb") as f:
data = f.read()
headers = {
"Content-Type": "text/plain", # required for PUT
"Content-Length": str(len(data)), # required for PUT
# For Azure Blob also add: "x-ms-blob-type": "BlockBlob"
}
resp = requests.put(put_url, data=data, headers=headers)
resp.raise_for_status()
# Verify by reading remote file back
print(remote_file.readString(0, 100))Handle file operations in an environment without writing to a file
In some scenarios, you may not want to create a local file before uploading content to the remote file system (for example, when pickling a model). In this case, use native Python libraries such as tempfile for operations that would normally require a local file. You then do not need to manage the local file after the upload.
The following example serializes a local_dict object as a YAML string, then uploads it to the remote file system as local_dict.yaml.
from tempfile import NamedTemporaryFile
import os
import yaml
# Create a local Python object
local_dict = {'id': '123', 'data': 42}
with NamedTemporaryFile() as temp_file:
# Create the temp file and write to it
dumped_yaml = yaml.dump(local_dict)
temp_file.write(bytes(dumped_yaml,encoding='utf8'))
temp_file.seek(0)
# Form the remote file URL
dest_url = c3.FileSystem.rootUrl() + 'local_dict.yaml'
# Write the content to the remote file location
c3.FileSystem.uploadFile(srcFile=temp_file, dstPath=dest_url)You can use any library that supports writing to a file to upload something to the FileSystem.