cbalci opened a new pull request #6465: URL: https://github.com/apache/incubator-pinot/pull/6465
## Description Adding some validation logic to ensure Dimension table sizes are properly constrained. This is a follow up to the [introduction of Dimension tables](https://github.com/apache/incubator-pinot/pull/6286) and https://github.com/apache/incubator-pinot/pull/6346 . In current implementation, dimension tables are fully loaded into heap memory, so this change is to prevent users from accidentally loading very large tables and disrupting the server. Changes include: - New controller config key: "controller.dimTable.maxSize" - Default value if no config is provided: DEFAULT_DIM_TABLE_MAX_SIZE = "200M" - Validation and enforcement logic in table creation flow (`PinotTableRestletResource`), such that the default max value is applied if no storage config exists for dimension tables. **Note:** I looked into enforcing this constraint in `TableConfigUtils.validate()` however it didn't work out due to package dependency issues. We need `ControllerConf` values for the validation but `TableConfigUtils` cannot depend on `ControllerConf` due to package structure. ## Testing - Controller tests are updated to include different scenarios. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
