javanna opened a new pull request, #16741: URL: https://github.com/apache/lucene/pull/16741
After merging #15012 to main, Lucene no longer strictly requires bumping the minimum supported major version when releasing a new major. That will still be necessary once breaking changes are introduced that require reindexing, but can be avoided when not necessary. This was discussed in the corresponding issue at #13797, yet while the change providing the infra for this important change was merged to main, the minimum supported version in main (future 11) remained 10. Several people asked if we could lower the minimum supported major version to 9 in main, so that Lucene 11 would be able to support reading and writing indices created by Lucene 9.x and effectively not require reindexing. I find that this would be a great improvement and would help users more quickly and less painfully migrate to Lucene 11 once it comes out. Effectively Lucene 11 would come with the minimum JDK bump requirement and breaking API changes, but no min major version bump. I did some research on the topic and this is possible today in main as no breaking changes were made at the codecs level. In practice, there does not seem to be any technical reason why we should prevent `IndexWriter` from opening indices created in 9.x in Lucene 11. From a cost and maintenance perspective, 9.x codecs are already in the codebase and would be shipped with Lucene 11 anyways, to allow users to read such indices using the existing expert API `DirectoryReader#open(IndexCommit commit, int minSupportedMajorVersion)`. It is a significant policy change that may cause confusion, as the compatibility policy will vary across major versions. Like we discussed in previous threads, reindexing will still be required in future major versions once necessary. Docs will need to be updated. I have not paid too much attention yet to javadocs `MIGRATE.md`. The change is easier than I initially thought, thanks to all the prep made with #15012 and consists of the following steps: - Adjust Version.MIN_SUPPORTED_MAJOR to 9 (was 10) - Adjust IndexWriterConfig#setIndexCreatedVersionMajor version checks (leftover LATEST.major - 1 check) - Formalize `MIN_BINARY_SUPPORTED_MAJOR` , as the minimum version that can be read using the expert read-only API, via the codecs that current Lucene ships with. This was 8.0 with Lucene 10.x, and becomes 9.0 with Lucene 11.0, as 8.x codecs are removed from main - Reinstate all the 9.x version constants as deprecated. - Remove all 9.x version from `unsupported_versions.txt` - Ensure all 9.x versions are covered in `versions.txt` - Rename all the test 9.x zip files, removing the unsupported prefix from their file name - Adjust test expectations: `TestIndexWriter`, `TestMoreTermsBackwardsCompatibility`, `TestMinSupportedMajorBackwardsCompatibility`, `TestEmptyIndexBackwardsCompatibility`, `TestDVUpdateBackwardsCompatibility`, `TestBinaryBackwardsCompatibility`, `TestBasicBackwardsCompatibility`, `TestAncientIndicesCompatibility`, `BackwardsCompatibilityTestBase` Note that given `MIN_SUPPORTED_MAJOR` and `MIN_BINARY_SUPPORTED_MAJOR ` align, write floor and expert-read floor are both 9, so there is no older major that is read-only-only. The expert API remains to cover older indices whose codecs are externally provided, and for a future scenario where the two min supported versions may diverge again. Lucene backwards compatibility tests look green against these changes. I was able to also get additional coverage from Elasticsearch tests run against this branch, that showed no compatibility issues. Are there more tests that we'd want to write to ensure this change is safe? Any additional checks to run? This change can easily be split into multiple PRs to ease reviews. Meanwhile, what is included in this PR is comprehensive and shows the proposed direction so we can start this discussion and collect feedback about it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
