[
https://issues.apache.org/jira/browse/LUCENE-9827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298412#comment-17298412
]
Robert Muir commented on LUCENE-9827:
-------------------------------------
You can find some info of what i did before to target this with benchmarks, on
the original issue adding tooDirty(): LUCENE-6183
Before that issue, we'd never be able to bulk-merge in practice because of
"alignment" issues: LUCENE-5646
Bulk merge as done with this tooDirty() stuff on LUCENE-6183 has some
limitations: e.g. it only happens for an append-only use-case, if there are any
deletes we trigger recompression. So we should keep the worst case in mind
going forwards too!
But you could address this problem in many totally different ways if we take a
step back. For example, we could re-compress every merge only with some
probability, which achieves the same goal (amortizing the recompression cost).
If there are deletes, maybe we can write some kind of "tombstones" and
bulk-copy existing data. Eventually it all gets merged away.
But these are not simple tradeoffs and could result in surprises, so the
tooDirty was trying to achieve the least surprise.
> Small segments are slower to merge due to stored fields since 8.7
> -----------------------------------------------------------------
>
> Key: LUCENE-9827
> URL: https://issues.apache.org/jira/browse/LUCENE-9827
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Adrien Grand
> Priority: Minor
> Attachments: total-merge-time-by-num-docs-on-small-segments.png
>
>
> [~dm] and [~dimitrisli] looked into an interesting case where indexing slowed
> down after upgrading to 8.7. After digging we identified that this was due to
> the merging of stored fields, which had become slower on average.
> This is due to changes to stored fields, which now have top-level blocks that
> are then split into sub-blocks and compressed using shared dictionaries (one
> dictionary per top-level block). As the top-level blocks are larger than they
> were before, segments are more likely to be considered "dirty" by the merging
> logic. Dirty segments are segments were 1% of the data or more consists of
> incomplete blocks. For large segments, the size of blocks doesn't really
> affect the dirtiness of segments: if you flush a segment that has 100 blocks
> or more, it will never be considered dirty as only the last block may be
> incomplete. But for small segments it does: for instance if your segment is
> only 10 blocks, it is very likely considered dirty given that the last block
> is always incomplete. And the fact that we increased the top-level block size
> means that segments that used to be considered clean might now be considered
> dirty.
> And indeed benchmarks reported that while large stored fields merges became
> slightly faster after upgrading to 8.7, the smaller merges actually became
> slower. See attached chart, which gives the total merge time as a function of
> the number of documents in the segment.
> I don't know how we can address this, this is a natural consequence of the
> larger block size, which is needed to achieve better compression ratios. But
> I wanted to open an issue about it in case someone has a bright idea how we
> could make things better.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
