msokolov commented on issue #12391: URL: https://github.com/apache/lucene/issues/12391#issuecomment-2086294536
> Since the writer logic is available in backward codecs for testing, I can still go ahead and override the codec write methods to work with old writers. But the path is not tested well in Lucene e.g. IndexWriter may not work with old codec version for writes, even for minor version upgrades. Although it is possible that IndexWriter would somehow stop being able to write an older version of a codec, that seems unlikely for a minor release. It is true that nothing enforces that that works. However running unit tests with your backwards-supporting codec should be enough to have confidence that it works. I think the answers to your yes/no questions are 1. Yes, 2. Yes. For 3, I'm not sure. It does seem like a difficult situation. I don't see how Lucene would support writing two index versions at the same time though. I think it sometimes happens that the backwards-codec implementations even drop support for writing, so it might not be a reliable solution to (2) in the general case. On Tue, Apr 30, 2024 at 10:31 AM itiyama ***@***.***> wrote: > Opensearch never went ahead with the proposed solution since it does not > work for major versions. I am wondering whether we should even rely on it > for minor versions and need your help on the same. Opensearch downgrades > are not seamless due to codec version compatibility issues during > deployments. > > *Current deployment process* > Transition from OpenSearch 2.x (Lucene 9.4) to 2.x+1 (Lucene 9.9) involves > moving all replicas followed by primaries. > Downgrading primaries is unsupported as there is no version of the > software that understands both old and new codec formats. > > *Proposed 2 phase deployment process* > Phase 1: Transition to OpenSearch 2.x+1 with Lucene 9.9, setting default > write version to 9_4. > Phase 2: Enable codec version 9_9 for OpenSearch 2.x+1. > > In this phased deployment, there is a version of software at every stage > which you can rollback to. > > *Challenges with supporting intermediate stage* > > 1. Old codecs are not supported > <https://github.com/apache/lucene/blob/branch_9_10/lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene90/Lucene90HnswVectorsFormat.java#L115> > out of the box for writes even for minor version upgrades. > 2. Since the writer logic is available in backward codecs for testing, > I can still go ahead and override the codec write methods to work with old > writers. But the path is not tested well in Lucene e.g. IndexWriter may not > work with old codec version for writes, even for minor version upgrades. > So, I am not comfortable in relying on this mechanism unless I am aware of > compatibility risks. > > *Questions* > > 1. Does Lucene officially support only the latest codec version for > write operations? > 2. Can I assess the compatibility risks associated with older codec > versions on new Lucene software by running the entire test suite with older > codec versions? Is this method sufficient for identifying potential issues? > 3. For applications relying on Lucene's segment replication model, and > lacking a stable software version for rollback, how can they address > deployment risks without independently verifying compatibility? > Alternatively, how can they manage deployments without a stable fallback > option, potentially risking downtime during rollbacks? Should Lucene > consider supporting this officially? > > *Remote store* > It is a special case for segment replication, so same problems exist. > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/lucene/issues/12391#issuecomment-2085490240>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAHHUQIYJM47NNONJLDTLATY76TLPAVCNFSM6AAAAAAZR5C5NSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBVGQ4TAMRUGA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
