Configure a Source Collection

In a data integration pipeline, a source collection reads one kind of source data, such as a folder of files or a database table. The source collection passes the data to the pipeline as Source Type rows. The programmatic quickstarts, such as Ingest Files from Amazon S3 Programmatically, create source collections that run on default settings. Review these defaults before a pipeline goes to production:

• A file under 5 MB processes as one unit.
• An extra source column is dropped without an error.
• An import keeps going whatever its error count.

To change these defaults, set configuration values on the source collection. You first decide which level a value applies at. Then you can split large files into chunks, control archiving, and reject files with unexpected columns. You can also describe nonstandard file formats to the parser, read Excel and custom formats, trace records to their source file, and raise the processing priority. "Fix common problems" lists the symptoms and fixes.

Copy link to this section

Prerequisites

Before you start, make sure you have the following:

• A source collection that already exists, such as the one you create in the Amazon S3 programmatic quickstart. The examples use file source collections such as ordersFileColl, ordersXlsxColl, and pipeOrdersColl, an Order target Type, and a source file named orders_2026_10.csv. Replace these names with your own.
• Access to the C3 AI Console of your C3 Agentic AI Platform application. You run every snippet on this page in the Console.

After you change a value, run the source collection again, as in the quickstart that created it, to see the effect.

Copy link to this section

Understand where configuration lives

A source collection has two kinds of settings, and you set them in different ways:

• Collection fields: part of the collection record itself, such as the change data capture (CDC) fields cdcOrder, fullSync, and cdcSourcesOverride that Load Data Incrementally describes. Set a collection field with its generated with* method, such as withFullSync(true), followed by upsert().
• Configuration values: every value in the rest of this page. Configuration values live in the collection's SourceCollection.Config object. Set a configuration value with setConfigValue, and remove the value with clearConfigValue.

var cfg = FileSourceCollection.forName('ordersFileColl').config();

cfg.setConfigValue('chunkSize', 500, ConfigOverride.APP);                // set
FileSourceCollection.forName('ordersFileColl').config().chunkSize;       // read
cfg.clearConfigValue('chunkSize', ConfigOverride.APP);                   // remove the APP value

setConfigValue stores the value and returns nothing. The cfg object you already hold keeps the old value, so call config() again to read the new one.

The ConfigOverride.APP argument scopes the value to your application. Administrators can also set values at the ENV (environment) or CLUSTER level. When several levels define a value, the most specific level wins: APP over ENV, and ENV over CLUSTER. Clearing a value at APP falls back to the ENV or CLUSTER value, then to the default. To see the value at each level, run cfg.allConfigValueOverrides('chunkSize').

To ship configuration values with your package, put them in the package file config/SourceCollection.Config/<collection name>.json. A collection with that name reads those values, and a value you set with setConfigValue at APP level overrides the package value. For example, with chunkSize 3 in the package file, setConfigValue('chunkSize', 9, ConfigOverride.APP) makes the collection use 9. You can also copy values from a running application into this file from Data Fusion. For the steps, see Promote and Version a Data Fusion Pipeline.

Most values take effect when the next run starts. A few, such as numErrorsToAbort, are read as each chunk imports, so a change can also affect later chunks of a run in progress.

Copy link to this section

Control chunking

Chunking splits a large file into chunks that process in parallel. Each chunk processes and reports on its own, so one bad chunk does not fail the whole file. Three values decide whether and how the platform splits a file:

• minByteSizeToChunk: the smallest file size, in bytes, that the platform chunks (default 5242880, which is 5 MB). A smaller file processes as one unit, whatever chunkSize says.
• chunkSize: the number of records per chunk. chunkSize has no default.
• chunkSizeInMb: the size of each chunk in megabytes (default 5). chunkSizeInMb applies only when chunkSize is not set.

chunkSize and chunkSizeInMb are alternatives, not combined limits. When chunkSize is set, the platform splits on record count only.

To see chunking on a small test file, lower both the threshold and the chunk size:

var cfg = FileSourceCollection.forName('ordersFileColl').config();
cfg.setConfigValue('chunkSize', 2, ConfigOverride.APP);
cfg.setConfigValue('minByteSizeToChunk', 1, ConfigOverride.APP);

With these values, a ten-row file produces five chunk-level SourceStatus records of two rows each, plus one file-level record for the whole file. Chunk records have typeIdent SOURCE:CHUNK, and the file record has typeIdent SOURCE.

doNotChunk turns chunking off and overrides chunkSize: with both set, the file processes as one chunk. Use doNotChunk when row order matters within a file or when the target entity Type cannot accept parallel writes.

Copy link to this section

Control archiving

The values in this section apply to file source collections. SQL source collections read external tables and archive nothing.

Each file source collection has an inbox, the folder it reads new files from, and an archive folder. By default, a processed file moves from the inbox to the archive, so later syncs pick up only new files. Three values change what the platform moves or writes:

• doNotArchiveSources: processed files stay in the inbox. Rows from the processed files still load.
• doNotArchiveFailedSources: the platform does not write records of failed sources to the collection's failed-sources location, which failedSourcesUrl() returns.
• doNotArchiveStatus: the platform does not record run status for the collection's sources. The platform creates no SourceStatus records and saves no status copy in the collection's statistics location, which statsUrl() returns. Status queries and the Data Monitoring page then show nothing for these runs.

Leave processed files in the inbox when another consumer reads the same folder. The inbox file count then no longer shows which files are pending, and you remove processed files yourself. SourceFile.removeWithData does not suit this cleanup, because it also deletes the entities loaded from the file.

Copy link to this section

Enforce strictness

By default, a file with an extra column loads: the platform drops the extra column, and the run reports completed with zero failures. To reject such files, set failOnExtraFields:

cfg.setConfigValue('failOnExtraFields', true, ConfigOverride.APP);

With failOnExtraFields set, the platform rejects the same file. No rows load, the SourceStatus state is rejected, and firstError names the extra column, the Source Type, and the header the platform read:

Invalid value path '<column><*>' at <position> for type <SourceType> and header <header>

Strict mode catches a renamed upstream column, which otherwise loads as an empty field with no error.

Two values relax checks for sources with inconsistent content. Both are off by default:

• ignoreBadContent: skips content that does not parse, and returns no rows or partial rows, instead of reporting an error.
• ignoreValueConversionErrors: sets a value that cannot convert to the field type to null instead of reporting an error. An example is a non-numeric string in a numeric field.

With either value set, the run status no longer reports parse or conversion errors, so check the loaded data instead.

numErrorsToAbort limits the errors in one import. Each source file, or each chunk of a chunked file, imports separately, and the platform counts its failed and rejected records. When the count reaches numErrorsToAbort, the platform stops that import. The default, -1, never stops an import: the platform processes every record and reports the failures when the import finishes. In production, set a small positive value so that a broken file stops early.

Copy link to this section

Parse and select source files

The values in this section apply to file source collections. Set these values when a file's format differs from the platform's parsing defaults, or to choose which files a collection reads:

Set these values with setConfigValue, like the other values on this page. To rewrite or classify content per line or per file, see the lineFilter and fileInfoOverride fields on FileSourceCollection. Each field takes a function that the platform runs on every line or every file.

Copy link to this section

Read Excel workbooks

A file source collection can read .xlsx workbooks. Legacy .xls, .xlsm, and .xlsb files are not supported. The collection reads every sheet with one Source Type, so define the Source Type with the union of the columns of all sheets. A column that a sheet lacks gets no value in that sheet's rows.

1. Set contentTypeOverride to the Excel content type on the collection, so that the platform parses the files as Excel workbooks:

   JavaScript

   Copy code to clipboard

   var cfg = FileSourceCollection.forName('ordersXlsxColl').config();
   cfg.setConfigValue('contentTypeOverride', 'application/vnd.c3.data+excel', ConfigOverride.APP);

2. To keep each row's sheet name, map the sourceFileSheet field in the transform to a field of the target entity Type, for example sheet: sourceFileSheet. The platform sets sourceFileSheet only while it imports the file, so a direct read of the file returns it as null.

3. (Optional) To process only some sheets, set sourceFileSheets to the list of sheet names:

   JavaScript

   Copy code to clipboard

   cfg.setConfigValue('sourceFileSheets', ['West'], ConfigOverride.APP);

4. Run the collection. With sourceFileSheets set to ['West'], only the rows of the West sheet load.

Two cell types read differently from the workbook view:

• An empty cell reads as null.
• A formula cell returns the value that Excel cached when the workbook was last saved. The platform does not recalculate formulas. A formula with no saved value, or with an error result, reads as empty.

Copy link to this section

Read a custom file format

For a format that no built-in parser reads, write a SerDeser Type that parses it, and point the collection at it with contentTypeOverride. The SerDeser creates objects of the collection's source Type. When the collection's source is a canonical Type, the files load into it with no Source Type and no transform into the canonical Type.

1. Declare the Type, mixing in SerDeser and implementing contentType and readObjs:

   Type

   Copy code to clipboard

   type PipeOrderSerDeser mixes SerDeser {
     contentType: ~ js-server
     readObjs: ~ js-server
   }

2. Implement the two methods. contentType returns the content type that selects this SerDeser. readObjs receives the file content and a read specification whose serType is the collection's source Type, and returns the parsed objects. This example reads lines of the form code|amount and skips lines that start with #:

   JavaScript

   Copy code to clipboard

   function contentType() {
     return 'application/x-pipe-orders';
   }
   
   function readObjs(content, spec) {
     var serType = spec && spec.serType ? spec.serType : OrderCanonical;
     var text = typeof content === 'string' ? content : content.readString();
     return text
       .split('\n')
       .filter(function (line) {
         return line.trim().length > 0 && line.charAt(0) !== '#';
       })
       .map(function (line) {
         var parts = line.split('|');
         return serType.make({ code: parts[0].trim(), amount: Number(parts[1]) });
       });
   }

3. After the package syncs, check that the platform finds the SerDeser by its content type. The call returns your Type:

   JavaScript

   Copy code to clipboard

   SerDeser.forContentType('application/x-pipe-orders', false);

4. Set the same content type on the collection, then run it:

   JavaScript

   Copy code to clipboard

   FileSourceCollection.forName('pipeOrdersColl').config()
     .setConfigValue('contentTypeOverride', 'application/x-pipe-orders', ConfigOverride.APP);

Replace OrderCanonical, code, and amount with your source Type and its fields.

Copy link to this section

Trace records to their source file

To record which file each target record came from, set includeMeta on the collection. includeMeta is off by default.

FileSourceCollection.forName('ordersFileColl').config()
  .setConfigValue('includeMeta', true, ConfigOverride.APP);

Records loaded after you set includeMeta carry the source file name in meta.sourceFile. Every target record then stores the file name, so the setting adds processing and storage. To turn on includeMeta for one run only, pass includeMeta: true in the DataIntegSpec that you give SourceFile.syncAll. To count the records that one file loaded, filter on meta.sourceFile:

Order.fetchCount({ filter: "meta.sourceFile=='orders_2026_10.csv'" });

Copy link to this section

Set processing priority

priority sets where the collection's work sits in the processing queues, relative to the work of other collections. Values fall in two bands: low priorities from 0 to 3000, and high priorities from 97000 to 100000. The platform reserves the values between 3000 and 97000, so do not assign them.

To give one pipeline precedence over the others, call setPriority with a value in the high band. setPriority stores the value and also reprioritizes work already in the queue, which setConfigValue alone does not do:

FileSourceCollection.forName('ordersFileColl').setPriority(98000);

To return to the default, call clearPriority(), which resets the queue priority to 300.

Copy link to this section

Fix common problems

Find the symptom you see, then apply the fix in the same row.

Copy link to this section

See also

• Integrate Data Programmatically
• Data Integration End-to-End Workflow
• Ingest Files from Amazon S3 Programmatically
• Select a Pipeline Execution Type
• Load Data Incrementally
• Monitor Your Data Pipeline
• Validate Source Data for Schema Drift
• Run BatchJobs Before and After a Pipeline