alessandrobenedetti commented on PR #12306: URL: https://github.com/apache/lucene/pull/12306#issuecomment-1553233035
> > Thus the change needn't be perfect in every possible way but needs to be a net positive. > > 💯 So lets "progress not perfection" on #12309 ? I still haven't heard a reason to justify the rush to a system property seven days after the last minor release. > > > Please elaborate on your "trappy" argument, Nick. > > A user **configures** this system property, indexes a 10Billion dimension vector. Goes to sleep. Lucene community wakes up, realizes empirical benchmarks and informed decisions matter, benchmarks the implementation and finds the **debt ceiling shouldn't be raised** and it gave the user community a cyanide pill wrapped as a tic tac, adds a different mechanism with strict restrictions to avoid a JVM crumble. User wakes up, upgrades lucene which now imposes this new smart safety mechanism, and the user now has to jump through stupid hoops to upgrade their incompatible index. THAT kind of trap. This kind of experience should be avoided at all cost, even if it's only likely to happen 1% of the time, because (to me) even giving a terrible experience to the 1% user is unacceptable. I get your point, but upgrades in general are tricky and care is necessary. If a new Lucene version won't support anymore a 10 Billion dimension vector for valid technical reasons(data structures, typing, etc) at that point it will be the responsibility of the user to: -1 not upgrade OR -2 Stick to some dimensionality reduction approach OR -3 abandon Lucene for another software With no configurability we currently leave only option 3 to our users anyway because we decided that 1024 is a golden limit for performance to be acceptable. The intention in configurability is to leave to the user the responsibility of deciding what performance is acceptable for his/her use case (at least my intention). I will spend more time on this later as a lot of good stuff has been added, but in relation to upgrades and back comaptibility I am not sure I see them as a critical blocker (happy to change my mind) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
