[
https://issues.apache.org/jira/browse/LUCENE-9286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17064712#comment-17064712
]
Dawid Weiss commented on LUCENE-9286:
-------------------------------------
Apologies for the delay. I'm still looking into this and trying to figure out
whether it's the automaton construction or something secondary in my own code.
We do FST traversals and cache arcs along the way: when doing so we do
arc.copyOf to freeze the state of an arc we have to return to (to follow the
target while still enumerating the remaining children). arc.copyOf can be
heavier now as it clones the underlying bitTable. Perhaps this could be just a
reference copy for read-only FSTs... this copying seems fairly heavy.
Still not sure whether this is the core of the problem. I'll get back to you.
> FST construction explodes memory in BitTable
> --------------------------------------------
>
> Key: LUCENE-9286
> URL: https://issues.apache.org/jira/browse/LUCENE-9286
> Project: Lucene - Core
> Issue Type: Bug
> Affects Versions: 8.5
> Reporter: Dawid Weiss
> Assignee: Dawid Weiss
> Priority: Major
> Attachments: screen-[1].png
>
>
> I see a dramatic increase in the amount of memory required for construction
> of (arguably large) automata. It currently OOMs with 8GB of memory consumed
> for bit tables. I am pretty sure this didn't require so much memory before
> (the automaton is ~50MB after construction).
> Something bad happened in between. Thoughts, [~broustant], [~sokolov]?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
