Re: Synonym filters memory usage

2019-10-02 Thread Dominique Bejean
Thank you for all your responses. Dominique Le lun. 30 sept. 2019 à 13:38, Erick Erickson a écrit : > Solr/Lucene _better_ not have a copy of the synonym map for every segment, > if so it’s a JIRA for sure. I’ve seen indexes with 100s of segments. With a > large synonym file it’d be terrible. >

Re: Synonym filters memory usage

2019-09-30 Thread Erick Erickson
Solr/Lucene _better_ not have a copy of the synonym map for every segment, if so it’s a JIRA for sure. I’ve seen indexes with 100s of segments. With a large synonym file it’d be terrible. I would be really, really, really surprised if this is the case. The Lucene people are very careful with me

Re: Synonym filters memory usage

2019-09-30 Thread Andrea Gazzarini
That sounds really strange to me. Segments are created gradually depending on changes applied to the index, while the Schema should have a completely different lifecycle, independent from that. If that is true, that would mean each time a new segment is created Solr would instantiate a new Sche

Re: Synonym filters memory usage

2019-09-30 Thread Bernd Fehling
Yes, I think so. While integrating a Thesaurus as synonyms.txt I saw massive memory usage. A heap dump and analysis with MemoryAnalyzer pointed out that the SynonymMap took 3 times a huge amount of memory, together with each opened index segment. Just try it and check that by yourself with heap du

Re: Synonym filters memory usage

2019-09-30 Thread Andrea Gazzarini
mmm, ok for the core but are you sure things in this case are working per-segment? I would expect a FilterFactory instance per index, initialized at schema loading time. On 30/09/2019 09:04, Bernd Fehling wrote: And I think this is per core per index segment. 2 cores per instance, each core w

Re: Synonym filters memory usage

2019-09-30 Thread Andrea Gazzarini
On 30/09/2019 09:04, Bernd Fehling wrote: And I think this is per core per index segment. 2 cores per instance, each core with 3 index segments, sums up to 6 times the 2 SynonymMaps. Results in 12 times SynonymMaps. Regards Bernd Am 30.09.19 um 08:41 schrieb Andrea Gazzarini:   Hi, looking

Re: Synonym filters memory usage

2019-09-30 Thread Bernd Fehling
And I think this is per core per index segment. 2 cores per instance, each core with 3 index segments, sums up to 6 times the 2 SynonymMaps. Results in 12 times SynonymMaps. Regards Bernd Am 30.09.19 um 08:41 schrieb Andrea Gazzarini: Hi, looking at the stateful nature of SynonymGraphFilter

Re: Synonym filters memory usage

2019-09-29 Thread Andrea Gazzarini
Hi, looking at the stateful nature of SynonymGraphFilter/FilterFactory classes, the answer should be 2 times (one time per type instance). The SynonymMap, which internally holds the synonyms table, is a private member of the filter factory and it is loaded each time the factory needs to create a t