Common Data Ingestion Issues
During the process of loading data into the C3 Agentic AI Platform, data ingestion errors are a common occurrence, especially when dealing with new datasets. Many of these errors fall into a few common categories. This section outlines some most frequent errors and provides guidance on how to diagnose and resolve them.
Invalid value path: SomeHeaderRowColumnName <*>
This error occurs when the platform cannot locate a corresponding target field in the Source type for a specific column name in the source file's header row. The * symbol highlights the problematic column in the error message. If the entire header row appears in the error message, it likely means that the platform could not identify the column delimiter correctly.
To resolve this invalid path issue
Add a field: Add a field to the Source Type that matches the column name exactly, including character case.
Use the @ser annotation: If the column name in the source file contains illegal characters or if you prefer a different field name, use the @ser annotation to define the mapping:
@ser(name="nameOfColumnInSourceFile")
fieldNamedTheWayILike: someDataTypeSkip extra fields: If the column is unnecessary for ingestion, set the skipExtraFields flag on the FileSourceCollection associated with the SourceFile. This ignores columns that do not map to a field in the Source type. However, be cautious, as this could lead to failing to ingest certain data, making it difficult to identify missing data due to column name changes or other issues.
Error on line n
This error is a general exception that indicates a problem with a specific row in the source file. However, the root cause is typically found in the inner exception within the stack trace. Often, this relates to a casting or formatting issue—for example, when data expected to be numeric (like "1239") includes values that cannot be cast to integers (like "123.41" or "31923 (Low)").
Resolve casting exceptions
Correct the source data: If possible, correct the data in the source system.
Write a transform: Implement a complex transform to parse the string and convert it into the appropriate data type.
Relax the target data type: Sometimes, changing the target data type to a more flexible one, such as a string, may be necessary.