[ 
https://issues.apache.org/jira/browse/LUCENE-9211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036806#comment-17036806
 ] 

Adrien Grand commented on LUCENE-9211:
--------------------------------------

I had a quick look at Juan's commit, there are things I like and things I have 
questions about. Since this PR is ready, or almost ready, I'd suggest merging 
this one first.

[~juan.duran] I saw that your commit tried to modify the current 
Lucene80DocValuesFormat. I'm a bit nervous about it because it makes it hard to 
spot any potential subtle difference in the on-disk format that would cause 
bugs, so I'd suggest creating a new Lucene85DocValuesFormat instead, even if it 
has the same ideas or even same on-disk format as the current 
Lucene80DocValuesFormat?

> Adding compression to BinaryDocValues storage
> ---------------------------------------------
>
>                 Key: LUCENE-9211
>                 URL: https://issues.apache.org/jira/browse/LUCENE-9211
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: core/codecs
>            Reporter: Mark Harwood
>            Assignee: Mark Harwood
>            Priority: Minor
>              Labels: pull-request-available
>
> While SortedSetDocValues can be used today to store identical values in a 
> compact form this is not effective for data with many unique values.
> The proposal is that BinaryDocValues should be stored in LZ4 compressed 
> blocks which can dramatically reduce disk storage costs in many cases. The 
> proposal is blocks of a number of documents are stored as a single compressed 
> blob along with metadata that records offsets where the original document 
> values can be found in the uncompressed content.
> There's a trade-off here between efficient compression (more docs-per-block = 
> better compression) and fast retrieval times (fewer docs-per-block = faster 
> read access for single values). A fixed block size of 32 docs seems like it 
> would be a reasonable compromise for most scenarios.
> A PR is up for review here [https://github.com/apache/lucene-solr/pull/1234]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to