[
https://issues.apache.org/jira/browse/LUCENE-9827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298925#comment-17298925
]
Robert Muir commented on LUCENE-9827:
-------------------------------------
So for the index sorting case it would be like an "insertion" against a block
as opposed to "deletion"?
Maybe the same mechanism could support both, allowing a limited number of
"patches" against the already-compressed-block.
For the shorter-term, I wonder if we can tweak the existing logic in the
meantime, adding a lower-bound on the number of dirty chunks before
recompressing. Let's start with the minimum possible value (N=2), and then go
from there? So it is just {{candidate.getNumDirtyChunks() > 1 && ...}}
So it avoids the case where we recompress 1 dirty chunk and potentially produce
another dirty chunk as a result. There must be at least 2 :)
something like this:
{code}
/**
* Returns true if we should recompress this reader, even though we could
bulk merge compressed
* data
*
* <p>The last chunk written for a segment is typically incomplete, so
without recompressing, in
* some worst-case situations (e.g. frequent reopen with tiny flushes), over
time the compression
* ratio can degrade. This is a safety switch.
*/
boolean tooDirty(Lucene90CompressingStoredFieldsReader candidate) {
// more than 1% dirty, or more than hard limit of 1024 dirty chunks
return candidate.getNumDirtyChunks() > 1024
|| (candidate.getNumDirtyChunks() > 1 && candidate.getNumDirtyDocs() *
100 > candidate.getNumDocs());
}
{code}
> Small segments are slower to merge due to stored fields since 8.7
> -----------------------------------------------------------------
>
> Key: LUCENE-9827
> URL: https://issues.apache.org/jira/browse/LUCENE-9827
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Adrien Grand
> Priority: Minor
> Attachments: total-merge-time-by-num-docs-on-small-segments.png
>
>
> [~dm] and [~dimitrisli] looked into an interesting case where indexing slowed
> down after upgrading to 8.7. After digging we identified that this was due to
> the merging of stored fields, which had become slower on average.
> This is due to changes to stored fields, which now have top-level blocks that
> are then split into sub-blocks and compressed using shared dictionaries (one
> dictionary per top-level block). As the top-level blocks are larger than they
> were before, segments are more likely to be considered "dirty" by the merging
> logic. Dirty segments are segments were 1% of the data or more consists of
> incomplete blocks. For large segments, the size of blocks doesn't really
> affect the dirtiness of segments: if you flush a segment that has 100 blocks
> or more, it will never be considered dirty as only the last block may be
> incomplete. But for small segments it does: for instance if your segment is
> only 10 blocks, it is very likely considered dirty given that the last block
> is always incomplete. And the fact that we increased the top-level block size
> means that segments that used to be considered clean might now be considered
> dirty.
> And indeed benchmarks reported that while large stored fields merges became
> slightly faster after upgrading to 8.7, the smaller merges actually became
> slower. See attached chart, which gives the total merge time as a function of
> the number of documents in the segment.
> I don't know how we can address this, this is a natural consequence of the
> larger block size, which is needed to achieve better compression ratios. But
> I wanted to open an issue about it in case someone has a bright idea how we
> could make things better.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
