MilindShyani commented on issue #13519: URL: https://github.com/apache/lucene/issues/13519#issuecomment-2229864587
Hi @benwtrent, I am trying to carefully check the math vs code. The first thing I noticed is that my definitions of dxs and dxq don't necessarily match with yours. That might be the origin for the sign error. I will uniformize the notation and get back. But meanwhile, in the latest code https://github.com/apache/lucene/compare/main...benwtrent:fix-8-bit?expand=1 I find that `float dxs = scale * dxc;` but scale is defined as 127/max-min. Shouldn't the 127 actually be 255? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
