tang-hi commented on issue #12396: URL: https://github.com/apache/lucene/issues/12396#issuecomment-1611775687
I have successfully implemented all encode methods in forutil while keeping the compression format unchanged. Here are the results. | Benchmark | Mode | Cnt | Score (ops/s) | Error (ops/s) | |---------------------------|------|-----|---------------|---------------| | Encode1 | thrpt| 15 | 30.81M | 7.76M | | vectorizedEncode1 | thrpt| 15 | 52.11M | 9.15M | | Encode2 | thrpt| 15 | 44.97M | 972.53K | | vectorizedEncode2 | thrpt| 15 | 84.76M | 11.56M | | Encode3 | thrpt| 15 | 39.16M | 389.41K | | vectorizedEncode3 | thrpt| 15 | 49.50M | 319.99K | | Encode4 | thrpt| 15 | 49.35M | 895.58K | | vectorizedEncode4 | thrpt| 15 | 84.05M | 1.10M | | Encode5 | thrpt| 15 | 35.39M | 1.76M | | vectorizedEncode5 | thrpt| 15 | 39.75M | 2.49M | | Encode6 | thrpt| 15 | 38.26M | 1.16M | | vectorizedEncode6 | thrpt| 15 | 52.07M | 2.29M | | Encode7 | thrpt| 15 | 37.20M | 556.95K | | vectorizedEncode7 | thrpt| 15 | 48.99M | 1.34M | | Encode8 | thrpt| 15 | 59.05M | 1.74M | | vectorizedEncode8 | thrpt| 15 | 95.48M | 7.30M | | Encode9 | thrpt| 15 | 24.64M | 135.16K | | vectorizedEncode9 | thrpt| 15 | 27.45M | 1.29M | | Encode10 | thrpt| 15 | 24.94M | 358.51K | | vectorizedEncode10 | thrpt| 15 | 27.22M | 2.40M | | Encode11 | thrpt| 15 | 25.11M | 819.30K | | vectorizedEncode11 | thrpt| 15 | 23.89M | 1.28M | | Encode12 | thrpt| 15 | 29.45M | 490.67K | | vectorizedEncode12 | thrpt| 15 | 41.90M | 2.30M | | Encode13 | thrpt| 15 | 24.38M | 226.59K | | vectorizedEncode13 | thrpt| 15 | 24.83M | 1.39M | | Encode14 | thrpt| 15 | 27.44M | 282.45K | | vectorizedEncode14 | thrpt| 15 | 36.28M | 3.62M | | Encode15 | thrpt| 15 | 27.64M | 504.77K | | vectorizedEncode15 | thrpt| 15 | 39.15M | 2.88M | | Encode16 | thrpt| 15 | 55.31M | 4.87M | | vectorizedEncode16 | thrpt| 15 | 84.89M | 7.14M | | Encode17 | thrpt| 15 | 16.27M | 152.66K | | vectorizedEncode17 | thrpt| 15 | 15.04M | 740.80K | | Encode18 | thrpt| 15 | 15.45M | 69.44K | | vectorizedEncode18 | thrpt| 15 | 15.49M | 654.85K | | Encode19 | thrpt| 15 | 12.67M | 119.34K | | vectorizedEncode19 | thrpt| 15 | 13.17M | 539.82K | | Encode20 | thrpt| 15 | 15.90M | 184.34K | | vectorizedEncode20 | thrpt| 15 | 16.03M | 1.29M | | Encode21 | thrpt| 15 | 15.95M | 87.33K | | vectorizedEncode21 | thrpt| 15 | 15.01M | 274.69K | | Encode22 | thrpt| 15 | 16.04M | 127.17K | | vectorizedEncode22 | thrpt| 15 | 15.34M | 404.39K | | Encode23 | thrpt| 15 | 15.62M | 180.99K | | vectorizedEncode23 | thrpt| 15 | 15.11M | 615.33K | | Encode24 | thrpt| 15 | 17.79M | 673.54K | | vectorizedEncode24 | thrpt| 15 | 22.53M | 2.13M | | Encode25 | thrpt| 15 | 15.42M | 203.42K | | vectorizedEncode25 | thrpt| 15 | 13.45M | 1.34M | | Encode26 | thrpt| 15 | 15.20M | 187.19K | | vectorizedEncode26 | thrpt| 15 | 14.91M | 402.92K | | Encode27 | thrpt| 15 | 14.98M | 81.71K | | vectorizedEncode27 | thrpt| 15 | 14.71M | 740.17K | | Encode28 | thrpt| 15 | 17.84M | 166.66K | | vectorizedEncode28 | thrpt| 15 | 20.75M | 836.14K | | Encode29 | thrpt| 15 | 14.66M | 177.74K | | vectorizedEncode29 | thrpt| 15 | 13.89M | 392.35 | | Encode30 | thrpt| 15 | 17.43M | 462.43K | | vectorizedEncode30 | thrpt| 15 | 23.42M | 2.45M | | Encode31 | thrpt| 15 | 18.49M | 280.37K | | vectorizedEncode31 | thrpt| 15 | 26.03M | 1.40M | | Encode32 | thrpt| 15 | 50.39M | 1.99M | | vectorizedEncode32 | thrpt| 15 | 38.64M | 51.39M | While vectorized code can be advantageous, there are a few drawbacks to consider. Firstly, if the vector bit size is 128bit, it may be slower than scalar code due to we compress the long value(64bit). To address this, it may be worthwhile to check if the user's host supports 256bit vector. If not, scalar code can be used for compression. Secondly, the code can become quite large due to the need for specific code for each bitPerValue. As a result, it can be comparable to writing assembly code, with approximately 6000+ lines of code required. I seek counsel from others regarding these two drawbacks. If they are deemed acceptable, I intend to submit a draft PR this weekend. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
