[ 
https://issues.apache.org/jira/browse/LUCENE-9981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17354542#comment-17354542
 ] 

Michael McCandless commented on LUCENE-9981:
--------------------------------------------

{quote}Caught one typo ("Automataon") in CHANGES.txt
{quote}
Woops, I'll fix.
{quote}One thing that bugs me is the int argument's "defaults". Currently it 
just says:
{quote}Set higher to allow more complex queries and lower to prevent memory 
exhaustion.
{quote}{quote}
We do provide a {{public static final int DEFAULT_DETERMINIZE_WORK_LIMIT = 
10000}}{{}} though it's not obvious :)  I'll update the javadocs to try to 
advertise that better.

There is some danger here that the {{10}} multiplier is "too much".  I picked 
this number as a rough conversion from maximum states in the final det machine, 
to total sum'd effort (which is just the sum of all powerset sizes of all DFA 
states iterated through), i.e. "on average" each det state consists of 10 NFA 
states.  If we make this too high, we risk ReDoS sort of attacks since we work 
too hard before giving up.  But if it's too low, then possibly legitimate 
regexps are incorrectly blocked.  I think 10 is OK?

> CompiledAutomaton.getCommonSuffix can be extraordinarily slow, even with 
> default maxDeterminizedStates limit
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-9981
>                 URL: https://issues.apache.org/jira/browse/LUCENE-9981
>             Project: Lucene - Core
>          Issue Type: Task
>            Reporter: Robert Muir
>            Priority: Major
>         Attachments: LUCENE-9981.patch, LUCENE-9981.patch, LUCENE-9981.patch, 
> LUCENE-9981.patch, LUCENE-9981_nfaprefix.patch, LUCENE-9981_test.patch, 
> three-repeats-reverse-det.png, three-repeats.png
>
>
> We have a {{maxDeterminizedStates = 10000}} limit designed to keep 
> regexp-type queries from blowing up. 
> But we have an adversary that will run for 268s on my laptop before hitting 
> exception, first reported here: 
> https://github.com/opensearch-project/OpenSearch/issues/687
> When I run the test and jstack the threads, this what I see:
> {noformat}
> "TEST-TestOpensearch687.testInteresting-seed#[4B9C20A027A9850C]" #15 prio=5 
> os_prio=0 cpu=56960.04ms elapsed=57.49s tid=0x00007fff7006ca80 nid=0x231c8 
> runnable  [0x00007fff8b7f0000]
>    java.lang.Thread.State: RUNNABLE
>       at 
> org.apache.lucene.util.automaton.SortedIntSet.decr(SortedIntSet.java:106)
>       at 
> org.apache.lucene.util.automaton.Operations.determinize(Operations.java:769)
>       at 
> org.apache.lucene.util.automaton.Operations.getCommonSuffixBytesRef(Operations.java:1155)
>       at 
> org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:247)
>       at 
> org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:104)
>       at 
> org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:82)
>       at org.apache.lucene.search.RegexpQuery.<init>(RegexpQuery.java:138)
>       at org.apache.lucene.search.RegexpQuery.<init>(RegexpQuery.java:114)
>       at org.apache.lucene.search.RegexpQuery.<init>(RegexpQuery.java:72)
>       at org.apache.lucene.search.RegexpQuery.<init>(RegexpQuery.java:62)
>       at 
> org.apache.lucene.TestOpensearch687.testInteresting(TestOpensearch687.java:42)
> {noformat}
> This is really sad, as {{getCommonSuffixBytesRef()}} is only supposed to be 
> an "up-front" optimization to make the actual subsequent terms-intensive part 
> of the query faster. But it makes the whole query run for nearly 5 minutes 
> before it does anything.
> So I definitely think we should improve {{getCommonSuffixBytesRef}} to be 
> more "best-effort". For example, it can reduce the lower bound to {{1000}} 
> and catch the exception like such:
> {code}
> try {
>    // this is slow, and just an opto anyway, so don't burn cycles on it for 
> some crazy worst-case.
>    // if we don't set this common suffix, the query will just run a bit 
> slower, that's all.
>    int limit = Math.min(1000, maxDeterminizedStates);
>    BytesRef suffix = Operations.getCommonSuffixBytesRef(binary, limit);
>    ... (setting commonSuffixRef)
> } catch (TooComplexTooDeterminizeException notWorthIt) {
>   commonSuffixRef = null;
> }
> {code}
> Another, maybe simpler option, is to just check that input state/transitions 
> accounts don't exceed some low limit N.
> Basically this opto is geared at stuff like leading wildcard query of "*foo". 
> By computing that the common suffix is "foo" we can spend less CPU in the 
> terms dictionary because we can first do a memcmp before having to run data 
> thru any finite state machine. It's really a microopt and we shouldn't be 
> spending whole seconds of cpu on it, ever.
> But I still don't quite understand how the current limits are giving the 
> behavior today, maybe there is a bigger issue and I don't want to shove 
> something under the rug.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to