Map Source Fields to Target Fields
The mapping step defines how source data is translated into the target data model. While the Source Schema describes the incoming structure, it does not determine how that data should populate the target. Mapping resolves differences in field names, data types, required fields, and any derived values needed by the target model. Without explicit mappings, the system cannot correctly construct the target entity, leading to structural mismatches or ingestion failures.
After creating a Transform, you define how fields from the Source Schema map to the Target Entity.
The screen is divided into three panels:
- Left panel — Source fields
- Center panel — Transformation logic
- Right panel — Target fields
Review the Source Fields
The left panel lists all fields defined in the Source Schema.
- Confirm the fields available for mapping.
- Use the search bar to filter out incoming data before transformation.
- Verify data types before mapping to ensure compatibility with the target structure.
Map Source Fields to Target Fields
The right panel displays the fields of the selected Target Entity.
For each target field:
- Enter a mapping expression in the corresponding row in the center panel.
You can:
- Reference a source field directly (for example: Status)
- Apply a function (for example: toUpper(Status))
- Combine multiple fields or apply conditional logic if needed
Each row represents how one target field will be populated.
Use the Functions Library (Optional)
Click Functions Library to:
- Browse available transformation functions
- Insert supported expressions
- Apply formatting, casting, or calculations
This is useful when performing normalization or derived field logic.
Preview the Mapping
Before saving:
- Click Preview Source to review incoming source data.
- Click Preview Transform to see how the mapped output will appear.
This helps validate:
- Field mappings
- Data type compatibility
- Expression correctness
When multiple source files are present, both preview options allow you to inspect the source and transformed output for each file.
Save the Transform
Once all required target fields are mapped and validated:
- Click Save.
The Transform node is now configured and ready for execution in the pipeline.
Best Practices
- Ensure all required target fields are mapped.
- Confirm data types align between source and target.
- Use preview before saving to avoid runtime errors.
- Keep expressions readable and avoid unnecessary complexity.
Preview the Transform Output
In the Transform configuration view, click Preview Transform.
The Transform Preview panel opens:
- The left pane lists the available source files (for example, events_20250607.csv, events_20250608.csv).
- The right pane displays the transformed output as a tabular preview.
Review the previewed data to verify:
- Field mappings are correct (for example, account IDs mapped consistently).
- Derived or transformed fields produce expected values.
- No unexpected null values or type mismatches appear.
This step validates the transform logic before persisting data into the target type.
After reviewing and validating, Click Save.
Common Mapping Errors
Misconfigurations in the Transform step typically surface during preview or pipeline execution. The following are common issues and how to resolve them.
1. Unmapped Required Target Fields
Issue:
A required field in the target entity or canonical is not mapped.
Impact:
The pipeline fails during validation or ingestion.
Resolution:
Ensure all required target fields are populated with either:
- A direct source field
- A derived expression
- A default/static value (if appropriate)
2. Data Type Mismatch
Issue:
The source field type does not match the target field type (for example, mapping a string to an integer field).
Impact:
Preview may fail or ingestion may throw conversion errors.
Resolution:
Use transformation functions to cast or convert values (for example, numeric conversion or date parsing).
3. Null or Missing Values
Issue:
Source data contains null values for fields required by the target.
Impact:
Runtime failures or rejected records.
Resolution:
Add conditional logic or default handling to ensure required fields are not null.
4. Incorrect Field References
Issue:
A field name is misspelled or does not exist in the Source Schema.
Impact:
Expression evaluation fails during preview or execution.
Resolution:
Verify field names in the Source panel and use autocomplete or filtering where available.
5. Complex Expressions Without Validation
Issue:
Overly complex or untested expressions are used without previewing results.
Impact:
Unexpected transformation output or execution errors.
Resolution:
Use Preview Transform to validate logic before saving.
Recommendation
Always preview transformations before saving and running the pipeline. Early validation reduces runtime errors and improves ingestion reliability.