gashutos commented on issue #12448: URL: https://github.com/apache/lucene/issues/12448#issuecomment-1645589833
@backslasht The overhead of `Index Sort` is very high. I ingested above 36 million documents with/without @timestamp indexsort and different is minimum 20% plus. Refer below numbers. Without Index sort on @timestamp ``` | Min Throughput | index-append | 182029 | docs/s | | Mean Throughput | index-append | 197051 | docs/s | | Median Throughput | index-append | 195665 | docs/s | | Max Throughput | index-append | 210468 | docs/s | | 50th percentile latency | index-append | 165.423 | ms | | 90th percentile latency | index-append | 231.446 | ms | | 99th percentile latency | index-append | 908.578 | ms | | 99.9th percentile latency | index-append | 8934.86 | ms | | 99.99th percentile latency | index-append | 10348.3 | ms | | 100th percentile latency | index-append | 10806.6 | ms | ``` With index sort on @timestamp on ascending order. ``` | Min Throughput | index-append | 141237 | docs/s | | Mean Throughput | index-append | 149861 | docs/s | | Median Throughput | index-append | 146907 | docs/s | | Max Throughput | index-append | 167086 | docs/s | | 50th percentile latency | index-append | 210.367 | ms | | 90th percentile latency | index-append | 315.659 | ms | | 99th percentile latency | index-append | 1458.79 | ms | | 99.9th percentile latency | index-append | 10476.3 | ms | | 99.99th percentile latency | index-append | 10963.3 | ms | | 100th percentile latency | index-append | 10994.6 | ms | ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
