deepthi912 opened a new pull request, #17039:
URL: https://github.com/apache/pinot/pull/17039
This PR adds a validation framework for Pinot transform functions used in
ingestion configs. It provides validation hooks for datatype checks in the
TransformFunction interface that individual functions can implement to validate
their configurations during table creation.
We can include validationMode in the transform function specification:
1. **LEGACY** Mode (Default)
Purpose: Backward compatibility - allows all existing type conversions
Behavior: No validation, accepts everything (current Pinot behavior)
Use Case: Existing tables that shouldn't break
2. **LENIENT** Mode (Recommended)
Purpose: Safe type conversions allowed
Behavior: Allows safe conversions like INT→LONG, FLOAT→DOUBLE, but blocks
unsafe ones like STRING→INT
Use Case: New tables that want some safety but flexibility
3. **STRICT** Mode
Purpose: Maximum type safety
Behavior: No automatic type conversions, exact type matching required
Use Case: Critical tables where type safety is paramount
```
{
"columnName": "processed_age",
"transformFunction": "CAST(age_string AS INT)",
"validationMode": "STRICT"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
