michaelwechner commented on PR #12306: URL: https://github.com/apache/lucene/pull/12306#issuecomment-1552960080
> > Just to make sure I understand the -1 correctly: The concern is that if a user can set the dimension by himself, because the Lucene version works well with this dimension at that time for this user, but let's assume that a future Lucene version will not work anymore or not work well enough anymore with the dimension set by the user, then this means the user cannot use this new Lucene version in the future. Is this the main concern? > > This is also my concern. > > I am less worried about properly implementing sysprops, catching security exceptions etc. :) We should not be adding this sysprop in the first place. > > Sure it is possible today to sneakily bypass Lucene's aKNN dimension limit check, with the unfortunate subclassing workaround Mike S posted on the dev list, but we should not be making it easier for users to fall into this dangerous and long-term trap (what this PR does). Rather, we should make it harder, e.g. by moving the limit down into the Codec itself where the HNSW algorithm determines how it scales. If the limit is implemented in the Codec, a user would have to swap a different Codec, or privately fork Lucene's default HNSW Codec component, making it very clear that they are entering "not supported by Lucene's back compat on its default Codec". Thanks for clarifying! I think it is great, that Lucene is providing back compatibility, but at the same time I think it should be possible to change the limit at your own risk (as mentioned by David Smiley also allow to configure it lower than MAX_DIMENSIONS) It would be great if we could accomplish this by moving the MAX_DIMENSIONS to the HNSW implementation and still give the user somehow the possibility to change MAX_DIMENSIONS. Does anything speak against this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
