siddharthteotia opened a new pull request #5503: URL: https://github.com/apache/incubator-pinot/pull/5503
## Description In PR https://github.com/apache/incubator-pinot/pull/5285, the raw index writer format was changed to use 8 byte offset for each chunk in the file header. The writer version was bumped to 3. This was done to support > 2GB indexes. The change was backward compatible to continue the support for reading existing/old segments using 4-byte offsets While there is no problem with the change, it prevents rollback. So if there is any orthogonal issue while rolling out a release, we can't rollback to older Pinot release since segments already generated with 8-byte offsets can't be read by old code. This PR introduces a config option to set the writer version (2 for using old 4-byte chunk offset and 3 which is latest for 8-byte chunk offset). This config option is **temporary**. To deploy a new version of our internal offline segment creation/push job, we would like this format to be disabled temporarily to have the flexibility of dealing with issues by rolling back the release at will. Once the roll-out is complete, this config option will be removed. ## Upgrade Notes Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion) No Does this PR fix a zero-downtime upgrade introduced earlier? No Does this PR otherwise need attention when creating release notes? Things to consider: - New configuration options - Deprecation of configurations - Signature changes to public methods/interfaces - New plugins added or old plugins removed * [ ] Yes (Please label this PR as **<code>release-notes</code>** and complete the section on Release Notes) ## Release Notes ## Documentation ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
