markharwood commented on a change in pull request #1543:
URL: https://github.com/apache/lucene-solr/pull/1543#discussion_r432467975
##########
File path:
lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java
##########
@@ -429,7 +429,18 @@ private void flushData() throws IOException {
}
}
maxUncompressedBlockLength = Math.max(maxUncompressedBlockLength,
uncompressedBlockLength);
- LZ4.compress(block, 0, uncompressedBlockLength, data, ht);
+
+ // Compression proved to hurt latency in some cases, so we're only
+ // enabling it on long inputs for now. Can we reduce the compression
+ // overhead and enable compression again, e.g. by building shared
+ // dictionaries that allow decompressing one value at once instead of
+ // forcing 32 values to be decompressed even when you only need one?
+ if (uncompressedBlockLength >= 32 * numDocsInCurrentBlock) {
+ LZ4.compress(block, 0, uncompressedBlockLength, data, highCompHt);
Review comment:
Maybe create a constant for this 32 ("MIN_COMPRESSABLE_FIELD_LENGTH" ?)
otherwise it might get confused with the 32 we happen to use for max num of
values in a block.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
