Import and Export Data

Use Export, a BatchJob provided by the C3 Agentic AI Platform, to export data to a file. Before exporting, you can perform some action on the Type instance that holds the data. You can export data as JSON, XML, or CSV. To load the exported files back into an application, use Import.

To use Export, start by specifying the details in an instance of BatchExportSpec. You can run the following examples on the windTurbine application included with the C3 Agentic AI Platform.

To use these examples as a tutorial, create an application with windTurbine as a dependency. Generate demo data by running WindTurbineDataGenerator.createDemoData(), then follow the examples as written.

Copy link to this section

Examples

The following examples show common uses of Export.

Copy link to this section

Export all the Type instance data

var exportSpec = BatchExportSpec.make({
   targetType: WindTurbine,
   numFiles: 100,
   fileUrlOrEncodedPathPrefix: "TestBatchExportWindTurbine"
});
var job = Export.startExport(exportSpec);

// Check job status
job.status();
// View File URLs when job is complete
job.get().fileList.urls

If job.get().fileList.urls returns an error or an empty array, verify that the exported Type has data. Then check the Export job run for errors.

To view the content in the files, use the following code example:

// Change index to view other files
var index = 0;
var urls = job.get().fileList.urls;
// Make a File, then read content from the file
// Second parameter in readString() is end character to limit content
// Otherwise, all text is read into memory (proceed with caution)
var content = FileSystem.makeFile(urls[index]).readString(0, 1000);

Copy link to this section

Export selected fields of a Type instance into a CSV file

When exporting data to a CSV, ensure that you provide contentType and csvHeader. If you do not provide csvHeader, the export job fails.

var exportSpec = BatchExportSpec.make({
   targetType: WindTurbine,
   numFiles: 100,
   fileUrlOrEncodedPathPrefix: "TestCsvExportWindTurbine",
   contentType: "csv",
   csvHeader: "id, location, manufacturer.id", // include fields to export
   filter: "contains(id, 'TURBINE')"
});
var job = Export.startExport(exportSpec);

// Check job status
job.status();
// View File URLs when job is complete
job.get().fileList.urls

For nested fields, de-nest them with dot notation in the csvHeader field. For example, to export certain fields from Meta such as created and createdBy, define the BatchExportSpec as shown:

var exportSpec = BatchExportSpec.make({
   targetType: "WindTurbine",
   numFiles: 100,
   fileUrlOrEncodedPathPrefix: "TestCsvExportWindTurbine",
   contentType: "csv",
   csvHeader: "id, meta.created, meta.createdBy", // include fields to export
   filter: "contains(id, 'TURBINE')"
});

If you only include meta in the csvHeader, the resulting field in the export is undefined.

Copy link to this section

Export results of evalMetrics as JSON

var expressions = ["GeneratorRotationSpeedDiff", "GearOilTemperatureDiff"];
var start = "2022-01-01";
var end = "2022-05-01";
var interval = "DAY";
var filter = "id == 'TURBINE-1'";
var evalSpec = EvalMetricsSpec.make({
   expressions: expressions,
   start: start,
   end: end,
   interval: interval
});
var actionParams = {"spec": evalSpec};
var exportSpec = BatchExportSpec.make({
   targetAction: PartiallyAppliedAction.make({typeName: "WindTurbine", actionName: "evalMetrics", args: actionParams}),
   filter: filter,
   numFiles: 100,
   contentType: "json",
   fileUrlOrEncodedPathPrefix: "TestBatchExportEvalMetrics"
});
var job = Export.startExport(exportSpec);

// Check job status
job.status();
// View File URLs when job is complete
job.get().fileList.urls

Copy link to this section

Specify BatchExportSpec for export

The following table describes a few key fields of BatchExportSpec for Export. For a full list of fields and their descriptions, run the command c3ShowType(BatchExportSpec) from Console.

Copy link to this section

Export job instance

The startExport() returns a job instance on which you can call a BatchJob method; for example, setMaxConcurrency(100):

var job =  Export.startExport(exportSpec);
job.setMaxConcurrency(100); // and other BatchJob functions

Copy link to this section

Import

Use Import to import data into the C3 Agentic AI Platform by reading files that Export wrote. The Import Type deserializes the files and then calls the callback action specified in the Import. Import is also a BatchJob.

To use Import, start by specifying the details in an instance of BatchImportSpec. See the following examples.

Copy link to this section

Import data of a Type instance

The following code works for any export file type (JSON, CSV, XML) that exists at the fileUrlOrEncodedPathPrefix specified.

var importSpec = BatchImportSpec.make({
   targetType: WindTurbine,
   fileUrlOrEncodedPathPrefix: "TestBatchExportWindTurbine"
});
var job = Import.startImport(importSpec);

// Check job status
job.status()

After the job completes, you can fetch on the targetType to ensure the data was imported correctly.

Copy link to this section

Specify BatchImportSpec for import

The following table describes a few key fields of BatchImportSpec for Import. For a full list of fields and their descriptions, run the command c3ShowType(BatchImportSpec) from Console.

Copy link to this section

Import job instance

The startImport() returns a job instance on which you can call a BatchJob method; for example, setMaxConcurrency(100):

var job =  Import.startImport(importSpec);
job.setMaxConcurrency(100); // and other BatchJob functions

Copy link to this section

Specify fileUrlOrEncodedPathPrefix

The parameter fileUrlOrEncodedPathPrefix controls the bucket and the URL to export and import data. The parameter accepts either a fully qualified URL or a path in the default file system.

If fileUrlOrEncodedPathPrefix is not a URL, Export and Import treat it as a prefix for a file. Export exports to, and Import imports from, the following path of the default file system:

// Root URL for default file system
var rootUrl = FileSystem.rootUrl();

// Format of file name
"<rootUrl>/<fileUrlOrEncodedPathPrefix>/<fileUrlOrEncodedPathPrefix>-<RANDOM_HASH>-<FILE_NUMBER>.<extension>"

The following are a few fileUrlOrEncodedPathPrefix examples with Amazon S3 as the default file system:

sampleTest

The path used to export to or import from is:

s3://<defaultBucketName>/<ENV_NAME>/<APP_NAME>/fs/sampleTest/sampleTest-<RANDOM_HASH>-<FILE_NUMBER>.<extension>

For example, a sample file could be:

s3://c3--platform/testenv/windturbine/fs/sampleTest/sampleTest-PGDMPB0CMS-0.json

sampleTest/somePrefix

The path used to export to or import from is:

s3://<defaultBucketName>/<ENV_NAME>/<APP_NAME>/fs/sampleTest/somePrefix-<RANDOM_HASH>-<FILE_NUMBER>.<extension>

For example, a sample file could be:

s3://c3--platform/testenv/windturbine/fs/sampleTest/somePrefix-PGDMPB0CMS-0.json

s3://c3--platform/testenv/windturbine/fs/sampleFolder/

This path puts all files in the folder it names, with no file name prefix. For example, a sample file could be:

s3://c3--platform/testenv/windturbine/fs/sampleFolder/PGDMPB0CMS-0.json

If the path has no / separator at the end, Export and Import treat its last part as a file prefix, not a folder. See the following example.

s3://c3--platform/testenv/windturbine/fs/sampleFolder/samplePrefix

This path puts all the files in the folder sampleFolder, and all files have the prefix samplePrefix. For example, a sample file could be:

s3://c3--platform/testenv/windturbine/fs/sampleFolder/samplePrefix-PGDMPB0CMS-0.json

Copy link to this section

See also

• Data Integration Overview
• Work With Files
• Configure SQL Sources