benwtrent commented on issue #16263: URL: https://github.com/apache/lucene/issues/16263#issuecomment-4718774043
> Your suggestion of encoding length information directly into the vector magnitude and using MIPS is interesting. The challenge is that it requires re-embedding or at least reprocessing the entire corpus and committing to a specific length-biasing strategy at indexing time. Your suggestion also requires this as Lucene would need to encode some information in the index, and it would be a much larger more significant change. Additionally, you don't need to re-embed, just multiply the vectors by length of doc. I worry that the API requirements for this type of change will be broad, complex, and little used. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
