[
https://issues.apache.org/jira/browse/LUCENE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034557#comment-17034557
]
Mark Harwood commented on LUCENE-9211:
--------------------------------------
Thanks Juan and David for your comments.
I ran the spatial.alg test and modified [this
line|[https://github.com/apache/lucene-solr/blob/master/lucene/benchmark/conf/spatial.alg#L31]]
to use the "composite" strategy in order to exercise the Binary DV storage. I
did four runs of master and PR 1234 and there wasn't a clear pattern of changes
in speed.
||Master read recs/s||PR 1234 read recs/s||
|875.66|884.96|
|869.94|841.75|
|823.38|853.97|
|842.11|878.73|
||Master write docs/s||PR 1234 write docs/s||
|7,688.46|{color:#000000}8,163.20{color}|
|8,223.35|{color:#000000}7,882.39{color}|
|7,381.71|{color:#000000}7,930.78{color}|
|8,385.32|7,925|
> Adding compression to BinaryDocValues storage
> ---------------------------------------------
>
> Key: LUCENE-9211
> URL: https://issues.apache.org/jira/browse/LUCENE-9211
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
> Reporter: Mark Harwood
> Assignee: Mark Harwood
> Priority: Minor
> Labels: pull-request-available
>
> While SortedSetDocValues can be used today to store identical values in a
> compact form this is not effective for data with many unique values.
> The proposal is that BinaryDocValues should be stored in LZ4 compressed
> blocks which can dramatically reduce disk storage costs in many cases. The
> proposal is blocks of a number of documents are stored as a single compressed
> blob along with metadata that records offsets where the original document
> values can be found in the uncompressed content.
> There's a trade-off here between efficient compression (more docs-per-block =
> better compression) and fast retrieval times (fewer docs-per-block = faster
> read access for single values). A fixed block size of 32 docs seems like it
> would be a reasonable compromise for most scenarios.
> A PR is up for review here [https://github.com/apache/lucene-solr/pull/1234]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]