[
https://issues.apache.org/jira/browse/LUCENE-10023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17377636#comment-17377636
]
Robert Muir commented on LUCENE-10023:
--------------------------------------
{quote}
In contrast, the current PR only runs analysis once, and avoids buffering
BytesRefs (i.e., via deepCopy() – also avoids creating disposable one-off
SortedSetDocValuesField objects), and instead sends BytesRefs directly to the
docValuesWriter.
{quote}
I'm not seeing this looking at your patch. I see duplicate inversion: e.g.
tokenstream consumed twice.
> Multi-token post-analysis DocValues
> -----------------------------------
>
> Key: LUCENE-10023
> URL: https://issues.apache.org/jira/browse/LUCENE-10023
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
> Reporter: Michael Gibney
> Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The single-token case for post-analysis DocValues is accounted for by
> {{Analyzer.normalize(...)}} (and formerly {{MultiTermAwareComponent}}); but
> there are cases where it would be desirable to have post-analysis DocValues
> based on multi-token fields.
> The main use cases that I can think of are variants of faceting/terms
> aggregation. I understand that this could be viewed as "trappy" for the naive
> "Moby Dick word cloud" case; but:
> # I think this can be supported fairly cleanly in Lucene
> # Explicit user configuration of this option would help prevent people
> shooting themselves in the foot
> # The current situation is arguably "trappy" as well; it just offloads the
> trappiness onto Lucene-external workarounds for systems/users that want to
> support this kind of behavior
> # Integrating this functionality directly in Lucene would afford consistency
> guarantees that present opportunities for future optimizations (e.g., shared
> Terms dictionary between indexed terms and DocValues).
> This issue proposes adding support for multi-token post-analysis DocValues
> directly to {{IndexingChain}}. The initial proposal involves extending the
> API to include {{IndexableFieldType.tokenDocValuesType()}} (in addition to
> existing {{IndexableFieldType.docValuesType()}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
