[ 
https://issues.apache.org/jira/browse/LUCENE-10449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17500204#comment-17500204
 ] 

Renaud Delbru commented on LUCENE-10449:
----------------------------------------

We will, thanks Robert.

> Performance regression due to LZ4 compression of TermsDict in 
> SortedSetDocValues
> --------------------------------------------------------------------------------
>
>                 Key: LUCENE-10449
>                 URL: https://issues.apache.org/jira/browse/LUCENE-10449
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/codecs
>    Affects Versions: 9.0
>            Reporter: Renaud Delbru
>            Priority: Major
>         Attachments: lucene-8.11-no-compression.png, lucene-9.png
>
>
> LUCENE-9843 removed the compression option for SortedSetDocValues term 
> dictionaries and enabled LZ4 compression all the time. This has quite an 
> impact on our workloads which heavily uses sorted set doc values. It may lead 
> to perf regression from 2x up to 5x. See samples below.
>  
> {code:java}
> ❯ times_tasks Elasticsearch 7.10.2 (Lucene 8.7) - no terms dict compression
> name                      type                        time_min time_max 
> time_p50 time_p90
> 7.10.2-22.6-SNAPSHOT.json total                       42       90       45    
>    66
> 7.10.2-22.6-SNAPSHOT.json SearchJoinRequest1          14       32       15    
>    18
> 7.10.2-22.6-SNAPSHOT.json SearchTaskBroadcastRequest2 23       53       27    
>    43
> ❯ times_tasks Elasticsearch 7.17.1 (Lucene 8.11) - with terms dict compression
> name                      type                        time_min time_max 
> time_p50 time_p90
> 7.17.0-27.1-SNAPSHOT.json total                       253      327      285   
>    310
> 7.17.0-27.1-SNAPSHOT.json SearchJoinRequest1          121      154      142   
>    152
> 7.17.0-27.1-SNAPSHOT.json SearchTaskBroadcastRequest2 122      173      140   
>    152
> ❯ times_tasks Elasticsearch 7.17.1 (Lucene 8.11) - lucene_default codec is 
> used to bypass the terms dict compression 
> name                        type                        time_min time_max 
> time_p50 time_p90
> 7.17.0-27.1-SNAPSHOT.json.2 total                       48       96       63  
>      75
> 7.17.0-27.1-SNAPSHOT.json.2 SearchJoinRequest1          19       44       25  
>      31
> 7.17.0-27.1-SNAPSHOT.json.2 SearchTaskBroadcastRequest2 23       42       29  
>      37
> ❯ times_tasks Elasticsearch 8.0 (Lucene 9.0) - with terms dict compression
> name                     type                        time_min time_max 
> time_p50 time_p90
> 8.0.0-28.0-SNAPSHOT.json total                       260      327      287    
>   313
> 8.0.0-28.0-SNAPSHOT.json SearchJoinRequest1          122      168      148    
>   158
> 8.0.0-28.0-SNAPSHOT.json SearchTaskBroadcastRequest2 123      165      139    
>   155
> {code}
> We can clearly see in the benchmark the impact of the terms dict compression 
> in our workload. Profiling the execution indicates that the bottleneck is the 
> {{{}LZ4.decompress{}}}. We have attached two screenshots of a flamegraph.
> The CPU time of the {{TermsDict.next}} method with Lucene 8.11 with no terms 
> dict compression is around 2 seconds, while the CPU time of the same method 
> in Lucene 9.0 is 12 seconds. This was measured on a small benchmark reading a 
> fixed number of times a sorted set doc values field. Each document is created 
> with a single keyword value that represents a UUID. 
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

Reply via email to