mcvsubbu commented on PR #10869: URL: https://github.com/apache/pinot/pull/10869#issuecomment-1581532821
> This doesn't seem to be the correct fix. > > The real problem is that we are using the table config from the ZK and schema from the segment file (we picked this way before because schema might not be stored in ZK when this task was implemented). Instead, we should also read the schema from the ZK so that schema is in-sync with the table config. You may read the schema in `PurgeTaskExecutor` and pass it into the `SegmentPurger`, then use it to create the `SegmentGeneratorConfig`. Minion purge task can purge data from old segments, which may have been built with an older schema (minus some columns). I thought the idea was to keep the old segments without the new column (and let the server auto-load default value into new column instead of baking it into the segment). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
