alessandrobenedetti commented on PR #12306:
URL: https://github.com/apache/lucene/pull/12306#issuecomment-1553233035

   > > Thus the change needn't be perfect in every possible way but needs to be 
a net positive.
   > 
   > 💯 So lets "progress not perfection" on #12309 ? I still haven't heard a 
reason to justify the rush to a system property seven days after the last minor 
release.
   > 
   > > Please elaborate on your "trappy" argument, Nick.
   > 
   > A user **configures** this system property, indexes a 10Billion dimension 
vector. Goes to sleep. Lucene community wakes up, realizes empirical benchmarks 
and informed decisions matter, benchmarks the implementation and finds the 
**debt ceiling shouldn't be raised** and it gave the user community a cyanide 
pill wrapped as a tic tac, adds a different mechanism with strict restrictions 
to avoid a JVM crumble. User wakes up, upgrades lucene which now imposes this 
new smart safety mechanism, and the user now has to jump through stupid hoops 
to upgrade their incompatible index. THAT kind of trap. This kind of experience 
should be avoided at all cost, even if it's only likely to happen 1% of the 
time, because (to me) even giving a terrible experience to the 1% user is 
unacceptable.
   
   I get your point, but upgrades in general are tricky and care is necessary.
   If a new Lucene version won't support anymore a 10 Billion dimension vector 
for valid technical reasons(data structures, typing, etc) at that point it will 
be the responsibility of the user to:
   -1 not upgrade OR
   -2 Stick to some dimensionality reduction approach OR
   -3 abandon Lucene for another software
   
   With no configurability we currently leave only option 3 to our users anyway 
because we decided that 1024 is a golden limit for performance to be acceptable.
   The intention in configurability is to leave to the user the responsibility 
of deciding what performance is acceptable for his/her use case (at least my 
intention).
   
   I will spend more time on this later as a lot of good stuff has been added, 
but in relation to upgrades and back comaptibility I am not sure I see them as 
a critical blocker (happy to change my mind)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

Reply via email to