I actually did that once as a test years ago, as well as support for "paging" through the wildcard terms with a starting offset, and it worked great.
One way to think of the feature is as the ability to "sample" the values of the wildcard. I mean, not all queries require absolute precision. Sometimes you just want to know whether "something" exists matching the pattern, or "generally" what the values look like. I think it would be worth a Jira. -- Jack Krupansky On Tue, Dec 30, 2014 at 6:16 AM, Modassar Ather <modather1...@gmail.com> wrote: > Hi, > > In the query having lots of wildcard can we put a limitation on number of > expansion of terms done against a wildcard token something like > maxBooleanClauses? > > Thanks, > Modassar > > On Mon, Dec 29, 2014 at 11:15 AM, Modassar Ather <modather1...@gmail.com> > wrote: > > > Thanks Jack for your suggestions. > > > > Regards, > > Modassar > > > > On Fri, Dec 26, 2014 at 6:04 PM, Jack Krupansky < > jack.krupan...@gmail.com> > > wrote: > > > >> Either you have too little RAM on each node or too much data on each > node. > >> > >> You may need to shard the data much more heavily so that the total work > on > >> a single query is distributed in parallel to more nodes, each node > having > >> a > >> much smaller amount of data to work on. > >> > >> First, always make sure that the entire Lucene index for each node fits > >> entirely in the system memory available for file system caching. > Otherwise > >> the queries will be I/O bound. Check your current queries to see if that > >> is > >> the case - are the nodes compute bound or I/O bound? If I/O bound, add > >> more > >> system memory until the queries are no longer I/O bound. If compute > bound, > >> shard more heavily until the query latency becomes acceptable. > >> > >> > >> > >> -- Jack Krupansky > >> > >> On Fri, Dec 26, 2014 at 1:02 AM, Modassar Ather <modather1...@gmail.com > > > >> wrote: > >> > >> > Thanks for your suggestions Erick. > >> > > >> > This may be one of those situations where you really have to > >> > push back at the users and understand why they insist on these > >> > kinds of queries. They must be very patient since it won't be > >> > very performant. That said, I've seen this pattern; there are > >> > certainly valid conditions under which response times can be > >> > many seconds if there are few users and they are doing very > >> > complex/expert-level things. > >> > > >> > We have tried educating the users but it did not work because they are > >> used > >> > to the old way. They feel that wildcard gives more control over the > >> results > >> > and may not fully understand stemming. > >> > > >> > Regards, > >> > Modassar > >> > > >> > On Thu, Dec 25, 2014 at 3:17 AM, Erick Erickson < > >> erickerick...@gmail.com> > >> > wrote: > >> > > >> > > There's no magic bullet here that I know of. If your requirements > >> > > are to support these huge, many-wildcard queries then you only > >> > > have a few choices: > >> > > > >> > > 1> redo the index. I was surprised at how little it bloated the > >> > > index as far as memory required is concerned to add ngrams. > >> > > The key here is that there really aren't very many unique terms. > >> > > If you use bigrams, then there are only maybe 36^2 distinct > >> > > combinations. (assuming English and including numbers). > >> > > > >> > > 2> Increase the number of shards, putting many fewer docs > >> > > on each shard. > >> > > > >> > > 3> give each shard a lot more memory. This isn't actually one > >> > > of my preferred solutions since GC issues may raise their ugly > >> > > heads here. > >> > > > >> > > 4> insert creative solution here. > >> > > > >> > > This may be one of those situations where you really have to > >> > > push back at the users and understand why they insist on these > >> > > kinds of queries. They must be very patient since it won't be > >> > > very performant. That said, I've seen this pattern; there are > >> > > certainly valid conditions under which response times can be > >> > > many seconds if there are few users and they are doing very > >> > > complex/expert-level things. > >> > > > >> > > Now, all that said, wildcards are often examples of poor habits > >> > > or habits learned in DB systems where the only hammer was > >> > > %whatever%. I've seen situations where users didn't > >> > > understand that Solr broke the input stream up into words. And > >> > > stemmed. And WordDelimiterFilterFactory did all the magic > >> > > for finding, say D.C. and DC. So it's worth looking at the actual > >> > > queries that are sent, perhaps talking to users and understanding > >> > > what they _want_ out of the system, then perhaps educating them > >> > > as to better ways to get what they want. > >> > > > >> > > Literally I've seen people insist on entering queries that > >> > > wildcarded _everything_ both pre and post wildcards because > >> > > they didn't realize that Solr tokenizes... > >> > > > >> > > Once you hit an OOM, all bets are off as Shawn outlined. > >> > > > >> > > Best, > >> > > Erick > >> > > > >> > > On Wed, Dec 24, 2014 at 1:57 AM, Modassar Ather < > >> modather1...@gmail.com> > >> > > wrote: > >> > > > Thanks for your response. > >> > > > > >> > > > How many items in the collection ? > >> > > > There are about 100 millions documents. > >> > > > > >> > > > How are configured cache in solrconfig.xml ? > >> > > > Each cache has size attribute as 128. > >> > > > > >> > > > Can you provide a sample of the query ? > >> > > > Does it fail immediately after solrcloud startup or after several > >> > hours ? > >> > > > It is a query with many terms(more than a thousand) and phrase > where > >> > > > phrases have many wildcards in it. > >> > > > Once such query is executed there are many zookeeper related > >> exceptions > >> > > and > >> > > > with a couple of such queries executed it goes for OutOfMemory. > >> > > > > >> > > > Thanks, > >> > > > Modassar > >> > > > > >> > > > > >> > > > On Wed, Dec 24, 2014 at 1:37 PM, Dominique Bejean < > >> > > dominique.bej...@eolya.fr > >> > > >> wrote: > >> > > > > >> > > >> And you didn’t give how many RAM on each servers ? > >> > > >> > >> > > >> 2014-12-24 8:17 GMT+01:00 Dominique Bejean < > >> dominique.bej...@eolya.fr > >> > >: > >> > > >> > >> > > >> > Modassar, > >> > > >> > > >> > > >> > How many items in the collection ? > >> > > >> > I mean how many documents per collection ? 1 million, 10 > >> millions, > >> > …? > >> > > >> > > >> > > >> > How are configured cache in solrconfig.xml ? > >> > > >> > What are the size attribute value for each cache ? > >> > > >> > > >> > > >> > Can you provide a sample of the query ? > >> > > >> > Does it fail immediately after solrcloud startup or after > several > >> > > hours ? > >> > > >> > > >> > > >> > Dominique > >> > > >> > > >> > > >> > 2014-12-24 6:20 GMT+01:00 Modassar Ather < > modather1...@gmail.com > >> >: > >> > > >> > > >> > > >> >> Thanks for your suggestions. > >> > > >> >> > >> > > >> >> I will look into the link provided. > >> > > >> >> http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap > >> > > >> >> > >> > > >> >> This is usually an anti-pattern. The very first thing > >> > > >> >> I'd be doing is trying to not do this. See ngrams for infix > >> > > >> >> queries, or shingles or ReverseWildcardFilterFactory or..... > >> > > >> >> > >> > > >> >> We cannot avoid multiple wildcards since that's is our user's > >> > > >> requirement. > >> > > >> >> We try to discourage it but the users insist on firing such > >> > queries. > >> > > >> Also, > >> > > >> >> ngrams etc. can be tried but our index is already huge and > >> ngrams > >> > may > >> > > >> >> further add lot to it. We are OK with such queries failing as > >> long > >> > as > >> > > >> >> other > >> > > >> >> queries are not affected. > >> > > >> >> > >> > > >> >> > >> > > >> >> Please find the details below. > >> > > >> >> > >> > > >> >> So, how many nodes in the cluster ? > >> > > >> >> There are total 4 nodes on the cluster. > >> > > >> >> > >> > > >> >> How many shards and replicas for the collection ? > >> > > >> >> There are 4 shards and no replica for any of them. > >> > > >> >> > >> > > >> >> How many items in the collection ? > >> > > >> >> If I understand the question correctly there are two > collection > >> on > >> > > each > >> > > >> >> node and there size on each node is approximately 190GB and > >> 130GB. > >> > > >> >> > >> > > >> >> What is the size of the index ? > >> > > >> >> There are two collection on each node and there size on each > >> node > >> > is > >> > > >> >> approximately 190GB and 130GB. > >> > > >> >> > >> > > >> >> How is updated the collection (frequency, how many items per > >> days, > >> > > what > >> > > >> is > >> > > >> >> your hard commit strategy) ? > >> > > >> >> It is an optimized index and read-only. There are no > >> inter-mediate > >> > > >> update. > >> > > >> >> > >> > > >> >> How are configured cache in solrconfig.xml ? > >> > > >> >> Filter cache, query result cache and document cache are > enabled. > >> > > >> >> Auto-warming is also done. > >> > > >> >> > >> > > >> >> Can you provide all other JVM parameters ? > >> > > >> >> -Xms20g -Xmx24g -XX:+UseConcMarkSweepGC > >> > > >> >> > >> > > >> >> Thanks again, > >> > > >> >> Modassar > >> > > >> >> > >> > > >> >> On Wed, Dec 24, 2014 at 2:29 AM, Dominique Bejean < > >> > > >> >> dominique.bej...@eolya.fr > >> > > >> >> > wrote: > >> > > >> >> > >> > > >> >> > Hi, > >> > > >> >> > > >> > > >> >> > I agree Erick it could be a good think to have more details > >> about > >> > > your > >> > > >> >> > configuration and collection. > >> > > >> >> > > >> > > >> >> > Your heap size is 32Gb. How many RAM on each servers ? > >> > > >> >> > > >> > > >> >> > By « 4 shard Solr cluster », you mean a 4 nodes Solr servers > >> or a > >> > > >> >> > collection with 4 shards ? > >> > > >> >> > > >> > > >> >> > So, how many nodes in the cluster ? > >> > > >> >> > How many shards and replicas for the collection ? > >> > > >> >> > How many items in the collection ? > >> > > >> >> > What is the size of the index ? > >> > > >> >> > How is updated the collection (frequency, how many items per > >> > days, > >> > > >> what > >> > > >> >> is > >> > > >> >> > your hard commit strategy) ? > >> > > >> >> > How are configured cache in solrconfig.xml ? > >> > > >> >> > Can you provide all other JVM parameters ? > >> > > >> >> > > >> > > >> >> > Regards > >> > > >> >> > > >> > > >> >> > Dominique > >> > > >> >> > > >> > > >> >> > 2014-12-23 17:50 GMT+01:00 Erick Erickson < > >> > erickerick...@gmail.com > >> > > >: > >> > > >> >> > > >> > > >> >> > > Second most important part of your message: > >> > > >> >> > > "When executing a huge query with many wildcards inside it > >> the > >> > > >> server" > >> > > >> >> > > > >> > > >> >> > > This is usually an anti-pattern. The very first thing > >> > > >> >> > > I'd be doing is trying to not do this. See ngrams for > infix > >> > > >> >> > > queries, or shingles or ReverseWildcardFilterFactory > or..... > >> > > >> >> > > > >> > > >> >> > > And if your corpus is very large with many unique terms > it's > >> > even > >> > > >> >> > > worse, but you haven't really told us about that yet. > >> > > >> >> > > > >> > > >> >> > > Best, > >> > > >> >> > > Erick > >> > > >> >> > > > >> > > >> >> > > On Tue, Dec 23, 2014 at 8:30 AM, Shawn Heisey < > >> > > apa...@elyograg.org> > >> > > >> >> > wrote: > >> > > >> >> > > > On 12/23/2014 4:34 AM, Modassar Ather wrote: > >> > > >> >> > > >> Hi, > >> > > >> >> > > >> > >> > > >> >> > > >> I have a setup of 4 shard Solr cluster with embedded > >> > > zookeeper on > >> > > >> >> one > >> > > >> >> > of > >> > > >> >> > > >> them. The zkClient time out is set to 30 seconds, -Xms > is > >> > 20g > >> > > and > >> > > >> >> -Xms > >> > > >> >> > > is > >> > > >> >> > > >> 24g. > >> > > >> >> > > >> When executing a huge query with many wildcards inside > it > >> > the > >> > > >> >> server > >> > > >> >> > > >> crashes and becomes non-responsive. Even the dashboard > >> does > >> > > not > >> > > >> >> > responds > >> > > >> >> > > >> and shows connection lost error. This requires me to > >> restart > >> > > the > >> > > >> >> > > servers. > >> > > >> >> > > > > >> > > >> >> > > > Here's the important part of your message: > >> > > >> >> > > > > >> > > >> >> > > > *Caused by: java.lang.OutOfMemoryError: Java heap space* > >> > > >> >> > > > > >> > > >> >> > > > > >> > > >> >> > > > Your heap is not big enough for what Solr has been asked > >> to > >> > do. > >> > > >> You > >> > > >> >> > > > need to either increase your heap size or change your > >> > > >> configuration > >> > > >> >> so > >> > > >> >> > > > that it uses less memory. > >> > > >> >> > > > > >> > > >> >> > > > > >> > http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap > >> > > >> >> > > > > >> > > >> >> > > > Most programs have pretty much undefined behavior when > an > >> > OOME > >> > > >> >> occurs. > >> > > >> >> > > > Lucene's IndexWriter has been hardened so that it tries > >> > > extremely > >> > > >> >> hard > >> > > >> >> > > > to avoid index corruption when OOME strikes, and I > believe > >> > that > >> > > >> >> works > >> > > >> >> > > > well enough that we can call it nearly bulletproof ... > but > >> > the > >> > > >> rest > >> > > >> >> of > >> > > >> >> > > > Lucene and Solr will make no guarantees. > >> > > >> >> > > > > >> > > >> >> > > > It's very difficult to have definable program behavior > >> when > >> > an > >> > > >> OOME > >> > > >> >> > > > happens, because you simply cannot know the precise > point > >> > > during > >> > > >> >> > program > >> > > >> >> > > > execution where it will happen, or what isn't going to > >> work > >> > > >> because > >> > > >> >> > Java > >> > > >> >> > > > did not have memory space to create an object. Going > >> > > unresponsive > >> > > >> >> is > >> > > >> >> > > > not surprising. > >> > > >> >> > > > > >> > > >> >> > > > If you can solve your heap problem, note that you may > run > >> > into > >> > > >> other > >> > > >> >> > > > performance issues discussed on the wiki page that I > >> linked. > >> > > >> >> > > > > >> > > >> >> > > > Thanks, > >> > > >> >> > > > Shawn > >> > > >> >> > > > > >> > > >> >> > > > >> > > >> >> > > >> > > >> >> > >> > > >> >> > >> > > >> >> On Wed, Dec 24, 2014 at 2:29 AM, Dominique Bejean < > >> > > >> >> dominique.bej...@eolya.fr > >> > > >> >> > wrote: > >> > > >> >> > >> > > >> >> > Hi, > >> > > >> >> > > >> > > >> >> > I agree Erick it could be a good think to have more details > >> about > >> > > your > >> > > >> >> > configuration and collection. > >> > > >> >> > > >> > > >> >> > Your heap size is 32Gb. How many RAM on each servers ? > >> > > >> >> > > >> > > >> >> > By « 4 shard Solr cluster », you mean a 4 nodes Solr servers > >> or a > >> > > >> >> > collection with 4 shards ? > >> > > >> >> > > >> > > >> >> > So, how many nodes in the cluster ? > >> > > >> >> > How many shards and replicas for the collection ? > >> > > >> >> > How many items in the collection ? > >> > > >> >> > What is the size of the index ? > >> > > >> >> > How is updated the collection (frequency, how many items per > >> > days, > >> > > >> what > >> > > >> >> is > >> > > >> >> > your hard commit strategy) ? > >> > > >> >> > How are configured cache in solrconfig.xml ? > >> > > >> >> > Can you provide all other JVM parameters ? > >> > > >> >> > > >> > > >> >> > Regards > >> > > >> >> > > >> > > >> >> > Dominique > >> > > >> >> > > >> > > >> >> > 2014-12-23 17:50 GMT+01:00 Erick Erickson < > >> > erickerick...@gmail.com > >> > > >: > >> > > >> >> > > >> > > >> >> > > Second most important part of your message: > >> > > >> >> > > "When executing a huge query with many wildcards inside it > >> the > >> > > >> server" > >> > > >> >> > > > >> > > >> >> > > This is usually an anti-pattern. The very first thing > >> > > >> >> > > I'd be doing is trying to not do this. See ngrams for > infix > >> > > >> >> > > queries, or shingles or ReverseWildcardFilterFactory > or..... > >> > > >> >> > > > >> > > >> >> > > And if your corpus is very large with many unique terms > it's > >> > even > >> > > >> >> > > worse, but you haven't really told us about that yet. > >> > > >> >> > > > >> > > >> >> > > Best, > >> > > >> >> > > Erick > >> > > >> >> > > > >> > > >> >> > > On Tue, Dec 23, 2014 at 8:30 AM, Shawn Heisey < > >> > > apa...@elyograg.org> > >> > > >> >> > wrote: > >> > > >> >> > > > On 12/23/2014 4:34 AM, Modassar Ather wrote: > >> > > >> >> > > >> Hi, > >> > > >> >> > > >> > >> > > >> >> > > >> I have a setup of 4 shard Solr cluster with embedded > >> > > zookeeper on > >> > > >> >> one > >> > > >> >> > of > >> > > >> >> > > >> them. The zkClient time out is set to 30 seconds, -Xms > is > >> > 20g > >> > > and > >> > > >> >> -Xms > >> > > >> >> > > is > >> > > >> >> > > >> 24g. > >> > > >> >> > > >> When executing a huge query with many wildcards inside > it > >> > the > >> > > >> >> server > >> > > >> >> > > >> crashes and becomes non-responsive. Even the dashboard > >> does > >> > > not > >> > > >> >> > responds > >> > > >> >> > > >> and shows connection lost error. This requires me to > >> restart > >> > > the > >> > > >> >> > > servers. > >> > > >> >> > > > > >> > > >> >> > > > Here's the important part of your message: > >> > > >> >> > > > > >> > > >> >> > > > *Caused by: java.lang.OutOfMemoryError: Java heap space* > >> > > >> >> > > > > >> > > >> >> > > > > >> > > >> >> > > > Your heap is not big enough for what Solr has been asked > >> to > >> > do. > >> > > >> You > >> > > >> >> > > > need to either increase your heap size or change your > >> > > >> configuration > >> > > >> >> so > >> > > >> >> > > > that it uses less memory. > >> > > >> >> > > > > >> > > >> >> > > > > >> > http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap > >> > > >> >> > > > > >> > > >> >> > > > Most programs have pretty much undefined behavior when > an > >> > OOME > >> > > >> >> occurs. > >> > > >> >> > > > Lucene's IndexWriter has been hardened so that it tries > >> > > extremely > >> > > >> >> hard > >> > > >> >> > > > to avoid index corruption when OOME strikes, and I > believe > >> > that > >> > > >> >> works > >> > > >> >> > > > well enough that we can call it nearly bulletproof ... > but > >> > the > >> > > >> rest > >> > > >> >> of > >> > > >> >> > > > Lucene and Solr will make no guarantees. > >> > > >> >> > > > > >> > > >> >> > > > It's very difficult to have definable program behavior > >> when > >> > an > >> > > >> OOME > >> > > >> >> > > > happens, because you simply cannot know the precise > point > >> > > during > >> > > >> >> > program > >> > > >> >> > > > execution where it will happen, or what isn't going to > >> work > >> > > >> because > >> > > >> >> > Java > >> > > >> >> > > > did not have memory space to create an object. Going > >> > > unresponsive > >> > > >> >> is > >> > > >> >> > > > not surprising. > >> > > >> >> > > > > >> > > >> >> > > > If you can solve your heap problem, note that you may > run > >> > into > >> > > >> other > >> > > >> >> > > > performance issues discussed on the wiki page that I > >> linked. > >> > > >> >> > > > > >> > > >> >> > > > Thanks, > >> > > >> >> > > > Shawn > >> > > >> >> > > > > >> > > >> >> > > > >> > > >> >> > > >> > > >> >> > >> > > >> > > >> > > >> > > >> > > >> > >> > > > >> > > >> > > > > >
