chishui commented on code in PR #16836:
URL: https://github.com/apache/lucene/pull/16836#discussion_r4310924535
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene90/LZ4WithPresetDictCompressionMode.java:
##########
@@ -144,6 +154,29 @@ public void decompress(DataInput in, int originalLength,
int offset, int length,
assert bytes.isValid();
}
+ /**
+ * Decompresses {@code decompressedLength} bytes of a block that holds
{@code blockLength}
+ * bytes. Compressible blocks are read into a byte[] first, since decoding
from a heap array is
+ * much faster than reading tokens one by one from the {@link DataInput}.
Incompressible blocks
+ * are mostly literals that are already copied in bulk, so they are
decoded from the {@link
+ * DataInput} to avoid copying them twice.
+ */
+ private int decompress(
+ DataInput in,
+ int compressedLength,
+ int blockLength,
+ int decompressedLength,
+ byte[] dest,
+ int dOff)
+ throws IOException {
+ if (compressedLength > blockLength - (blockLength >>> 3)) {
+ return LZ4.decompress(in, decompressedLength, dest, dOff);
+ }
+ compressed = ArrayUtil.growNoCopy(compressed, compressedLength);
Review Comment:
`compressed` adds around `15kb` memory which is roughly 8–15% increase for
the class. It's held by storeFields() which is per segment. In situation like
normal fetch, segment is handled one by one, so it shouldn't be a problem. But
in situation like segments are processed in parallel, your concern is valid
that the added memory could times segment count.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
