drempapis opened a new pull request, #15937: URL: https://github.com/apache/lucene/pull/15937
### Problem Lucene's `Operations.determinize` uses powerset construction to convert an NFA into a DFA. For patterns with combinatorial structure (e.g. *a*b*c*d*e*f*), this can cause exponential blowup in the number of DFA states, each carrying its own FrozenIntSet snapshot, HashMap entry, and backing arrays. The existing `workLimit` parameter bounds `CPU` effort but provides no memory signal, the JVM can `OOM` long before the work limit is reached. There is currently no way for callers to observe or react to memory growth during determinization. ### Update This PR adds a new overload to Lucene's determinization API ``` public static Automaton determinize(Automaton a, int workLimit, LongConsumer progressCallback) ``` When a non-null callback is provided, Lucene periodically invokes it with an accumulated estimate of bytes allocated since the last report. The caller can use this signal to enforce memory policies (e.g. throw a circuit-breaker exception to abort determinization) The estimate is built from the allocations directly attributable to each newly discovered DFA state during powerset construction. Rather than invoking the callback on every new state (which would add overhead proportional to state count), estimates are accumulated and reported in chunks once a configurable byte threshold is crossed. Any remainder is flushed once at the end -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
