[
https://issues.apache.org/jira/browse/LUCENE-9827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17302307#comment-17302307
]
Daniel Mitterdorfer commented on LUCENE-9827:
---------------------------------------------
I ran a benchmark with a logging workload against Elasticsearch 7.10.1, which
uses Lucene 8.7.0 (baseline) and a modified version including the patch in
LUCENE-9827 (contender). The benchmark ran on the same hardware. Below are the
results:
||Metric||Task||Baseline||Contender||Diff||Unit||
|Cumulative indexing time of primary shards| |757.3|765.625|8.32562|min|
|Cumulative merge time of primary shards| |638.182|625.457|-12.7256|min|
|Cumulative merge count of primary shards| |8757|8756|-1| |
|Cumulative merge throttle time of primary shards| |153.09|121.933|-31.1565|min|
|Cumulative refresh time of primary shards| |134.334|138.532|4.19763|min|
|Cumulative refresh count of primary shards| |68581|68637|56| |
|Cumulative flush time of primary shards| |3.75047|3.84487|0.0944|min|
|Cumulative flush count of primary shards| |422|422|0| |
|Store size| |55.8773|56.5012|0.62398|GB|
|Translog size| |18.0484|18.5167|0.46833|GB|
|Segment count| |879|858|-21| |
|Min Throughput|bulk-index|416.335|540.285|123.949|docs/s|
|Mean Throughput|bulk-index|12327.8|12224|-103.831|docs/s|
|Median Throughput|bulk-index|12309|12187.6|-121.493|docs/s|
|Max Throughput|bulk-index|13134.7|13310.8|176.101|docs/s|
|50th percentile service time|bulk-index|1161.87|1170.35|8.47977|ms|
|90th percentile service time|bulk-index|1782.84|1791.81|8.96847|ms|
|99th percentile service time|bulk-index|2602.66|2598.91|-3.75005|ms|
|99.9th percentile service time|bulk-index|3306.02|3275.2|-30.8211|ms|
|99.99th percentile service time|bulk-index|3889.09|3811.8|-77.2825|ms|
|100th percentile service time|bulk-index|5967.85|5530.39|-437.466|ms|
|error rate|bulk-index|0|0|0|%|
We've also analyzed merge behavior as before using trace logging.
Compared to Lucene 8.6.2 we see more merges happening regardless of LUCENE-9827
(which is expected). The data are grouped by the number of documents in the
candidate segment.
!merge-count-by-num-docs.png!
We can also see that the patch biases Lucene now towards optimized merges (it's
the middle bar in both groups):
!merge-type-by-version.png!
Finally, we can see that the time that we spent in merges on the smallest
buckets of segments has been reduced drastically but is gradually increasing
with for candidate segments below 100.000 documents:
!total-merge-time-by-num-docs.png!
_Note: We define "time spent in merges" as the time period between entering the
respective branch in the code and exiting it. See [^log-and-lucene-9827.patch]
for details._
All in all it seems to me that in our scenario merging in Lucene behaves now
more predictable with a slightly lower cumulative merge time. From a
macro-perspective the performance characteristics are similar, but I'd need to
run more iterations of the benchmark to better quantify run-to-run variation
though.
> Small segments are slower to merge due to stored fields since 8.7
> -----------------------------------------------------------------
>
> Key: LUCENE-9827
> URL: https://issues.apache.org/jira/browse/LUCENE-9827
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Adrien Grand
> Priority: Minor
> Attachments: log-and-lucene-9827.patch, merge-count-by-num-docs.png,
> merge-type-by-version.png,
> total-merge-time-by-num-docs-on-small-segments.png,
> total-merge-time-by-num-docs.png
>
>
> [~dm] and [~dimitrisli] looked into an interesting case where indexing slowed
> down after upgrading to 8.7. After digging we identified that this was due to
> the merging of stored fields, which had become slower on average.
> This is due to changes to stored fields, which now have top-level blocks that
> are then split into sub-blocks and compressed using shared dictionaries (one
> dictionary per top-level block). As the top-level blocks are larger than they
> were before, segments are more likely to be considered "dirty" by the merging
> logic. Dirty segments are segments were 1% of the data or more consists of
> incomplete blocks. For large segments, the size of blocks doesn't really
> affect the dirtiness of segments: if you flush a segment that has 100 blocks
> or more, it will never be considered dirty as only the last block may be
> incomplete. But for small segments it does: for instance if your segment is
> only 10 blocks, it is very likely considered dirty given that the last block
> is always incomplete. And the fact that we increased the top-level block size
> means that segments that used to be considered clean might now be considered
> dirty.
> And indeed benchmarks reported that while large stored fields merges became
> slightly faster after upgrading to 8.7, the smaller merges actually became
> slower. See attached chart, which gives the total merge time as a function of
> the number of documents in the segment.
> I don't know how we can address this, this is a natural consequence of the
> larger block size, which is needed to achieve better compression ratios. But
> I wanted to open an issue about it in case someone has a bright idea how we
> could make things better.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
