performance implications on using lots of values in fq
I have documents in SOLR such that each document contains one to many points (latitude and longitudes). Currently we store the multiple points for a given document in the db and query the db to find all of the document ids around a given point first. Once we have the list of ids, we populate the fq with those ids and the q value and send that off to SOLR to do a search. In the "longest" query to SOLR we're populating about 450 ids into the fq parameter at this time. I was wondering if anyone knows the performance implications of passing so many ids into the fq and when it would potentially be a problem for SOLR? Currently the query passing in 450 ids is not a problem at all and returns in less than a second. Thanks. -- View this message in context: http://www.nabble.com/performance-implications-on-using-lots-of-values-in-fq-tp18617397p18617397.html Sent from the Solr - User mailing list archive at Nabble.com.
NPE on auto-warming and out of memory issues
We are experiencing OOM issues with a SOLR index that has about 12G of indexed data with 2GB allocated to the JVM. We first see these type of messages in the log: Oct 18, 2007 10:25:00 AM org.apache.solr.core.SolrException log SEVERE: Error during auto-warming of key:+(search_place_type:citi search_place_type:attract search_place_type:airport search_place_type:univers):java.lang .OutOfMemoryError: Java heap space Oct 18, 2007 10:25:07 AM org.apache.solr.core.SolrException log SEVERE: Error during auto-warming of key:+search_place_type:busi:java.lang.OutOfMemoryError: Java heap space After a log of continuing messages like the ones above we'll see log series of messages like this: Oct 18, 2007 10:55:58 AM org.apache.solr.core.SolrException log SEVERE: Error during auto-warming of key:[EMAIL PROTECTED]:java.lang.NullPointerException at org.apache.lucene.index.Term.compareTo(Term.java:91) at org.apache.lucene.index.TermInfosReader.getIndexOffset(TermInfosReader.java:112) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:147) at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:342) at org.apache.lucene.index.MultiReader.docFreq(MultiReader.java:220) at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:87) at org.apache.lucene.search.Similarity.idf(Similarity.java:459) at org.apache.lucene.search.TermQuery$TermWeight.(TermQuery.java:44) at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:145) at org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.(DisjunctionMaxQuery.java:99) at org.apache.lucene.search.DisjunctionMaxQuery.createWeight(DisjunctionMaxQuery.java:161) at org.apache.lucene.search.BooleanQuery$BooleanWeight.(BooleanQuery.java:187) at org.apache.lucene.search.BooleanQuery$BooleanWeight2.(BooleanQuery.java:342) at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:384) at org.apache.lucene.search.Query.weight(Query.java:95) at org.apache.lucene.search.Searcher.createWeight(Searcher.java:171) at org.apache.lucene.search.Searcher.search(Searcher.java:118) at org.apache.lucene.search.Searcher.search(Searcher.java:97) at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:888) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:805) at org.apache.solr.search.SolrIndexSearcher.access$100(SolrIndexSearcher.java:60) at org.apache.solr.search.SolrIndexSearcher$2.regenerateItem(SolrIndexSearcher.java:251) at org.apache.solr.search.LRUCache.warm(LRUCache.java:193) at org.apache.solr.search.SolrIndexSearcher.warm(SolrIndexSearcher.java:1385) at org.apache.solr.core.SolrCore$1.call(SolrCore.java:488) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269) at java.util.concurrent.FutureTask.run(FutureTask.java:123) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) at java.lang.Thread.run(Thread.java:595) We have our configuration set for autowarmCount to: It appears that autowarming has something to do with our OOM. I've read that if you set the autowarmCount to 0 or a lower value then the first request after a commit may take some time.Definitely willing to try setting the autowarmCount to a lower value. Does anyone have any other ideas to help with this autowarming issue? Thanks. -- View this message in context: http://www.nabble.com/NPE-on-auto-warming-and-out-of-memory-issues-tf4654164.html#a13297439 Sent from the Solr - User mailing list archive at Nabble.com.
restricting search to a set of documents
I need to perform a search against a limited set of documents. I have the set of document ids, but was wondering what is the best way to formulate the query to SOLR? -- View this message in context: http://www.nabble.com/restricting-search-to-a-set-of-documents-tf4767801.html#a13637479 Sent from the Solr - User mailing list archive at Nabble.com.
querying with two words returns less results when ORing terms
I'm searching using query of 'food' and get back 572 results. When I do a search for 'food drink' I am expecting more results since it's defaulting to ORing the search terms, however, I now get back 6 results. I'm sure I'm doing something stupid, but I don't know what for sure. I am using a DismaxQueryHandler for this and boosting certain fields. I've included some of the debug info for the 'food drink' results.Thanks. food drink food drink +((DisjunctionMaxQuery((summary:food^2.0 | places:food^2.5 | tags:food^10.0 | content:food | name:food^15.0 | author:food^10.0 | locations:food^5.0)~0.01) DisjunctionMaxQuery((summary:drink^2.0 | places:drink^2.5 | tags:drink^10.0 | content:drink | name:drink^15.0 | author:drink^10.0 | locations:drink^5.0)~0.01))~2) DisjunctionMaxQuery((summary:"food drink"~2^2.0 | places:"food drink"~2^2.5 | tags:"food drink"~2^10.0 | content:"food drink"~2 | name:"food drink"~2^15.0 | author:"food drink"~2^10.0 | locations:"food drink"~2^5.0)~0.01) -- View this message in context: http://www.nabble.com/querying-with-two-words-returns-less-results-when-ORing-terms-tp16942461p16942461.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: querying with two words returns less results when ORing terms
It was set to AND as the default so I changed it to restarted SOLR but still seeing the same results. The debug output looks the same as well. Alok K. Dhir wrote: > > check for > > > > in schema.xml > > On Apr 28, 2008, at 12:16 PM, briand wrote: > >> >> I'm searching using query of 'food' and get back 572 results. When >> I do a >> search for 'food drink' I am expecting more results since it's >> defaulting to >> ORing the search terms, however, I now get back 6 results. I'm >> sure I'm >> doing something stupid, but I don't know what for sure. I am using a >> DismaxQueryHandler for this and boosting certain fields. I've >> included >> some of the debug info for the 'food drink' results.Thanks. >> >> food drink >> >> food drink >> >> >> +((DisjunctionMaxQuery((summary:food^2.0 | places:food^2.5 | >> tags:food^10.0 >> | content:food | name:food^15.0 | author:food^10.0 | >> locations:food^5.0)~0.01) DisjunctionMaxQuery((summary:drink^2.0 | >> places:drink^2.5 | tags:drink^10.0 | content:drink | name:drink^15.0 | >> author:drink^10.0 | locations:drink^5.0)~0.01))~2) >> DisjunctionMaxQuery((summary:"food drink"~2^2.0 | places:"food >> drink"~2^2.5 >> | tags:"food drink"~2^10.0 | content:"food drink"~2 | name:"food >> drink"~2^15.0 | author:"food drink"~2^10.0 | locations:"food >> drink"~2^5.0)~0.01) >> >> >> -- >> View this message in context: >> http://www.nabble.com/querying-with-two-words-returns-less-results-when-ORing-terms-tp16942461p16942461.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> > > > -- View this message in context: http://www.nabble.com/querying-with-two-words-returns-less-results-when-ORing-terms-tp16942461p16942834.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: querying with two words returns less results when ORing terms
Yep setting mm=1 did the trick for us. Thanks! Howard Lee-2 wrote: > > I don't think the defaultOperator applies to Dismax in the same way as the > Standard Request Handler. The Dismax handler uses the mm (Minimum 'Should' > Match) parameter to specify how many words should match the query. You > may need to change this to get it working the way you want it. > > http://wiki.apache.org/solr/DisMaxRequestHandler?highlight=%28CategorySolrRequestHandler%29%7C%28%28CategorySolrRequestHandler%29%29#head-6c5fe41d68f3910ed544311435393f5727408e61 > > Howard > > 2008/4/28 briand <[EMAIL PROTECTED]>: > >> >> I'm searching using query of 'food' and get back 572 results. When I do >> a >> search for 'food drink' I am expecting more results since it's defaulting >> to >> ORing the search terms, however, I now get back 6 results. I'm sure I'm >> doing something stupid, but I don't know what for sure. I am using a >> DismaxQueryHandler for this and boosting certain fields. I've included >> some of the debug info for the 'food drink' results.Thanks. >> >> food drink >> >> food drink >> >> >> +((DisjunctionMaxQuery((summary:food^2.0 | places:food^2.5 | >> tags:food^10.0 >> | content:food | name:food^15.0 | author:food^10.0 | >> locations:food^5.0)~0.01) DisjunctionMaxQuery((summary:drink^2.0 | >> places:drink^2.5 | tags:drink^10.0 | content:drink | name:drink^15.0 | >> author:drink^10.0 | locations:drink^5.0)~0.01))~2) >> DisjunctionMaxQuery((summary:"food drink"~2^2.0 | places:"food >> drink"~2^2.5 >> | tags:"food drink"~2^10.0 | content:"food drink"~2 | name:"food >> drink"~2^15.0 | author:"food drink"~2^10.0 | locations:"food >> drink"~2^5.0)~0.01) >> >> >> -- >> View this message in context: >> http://www.nabble.com/querying-with-two-words-returns-less-results-when-ORing-terms-tp16942461p16942461.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > -- View this message in context: http://www.nabble.com/querying-with-two-words-returns-less-results-when-ORing-terms-tp16942461p16945652.html Sent from the Solr - User mailing list archive at Nabble.com.
stemming the synonyms
I'm using the synonyms text file such that when you enter travel you find all things related to vacation. However, when I enter in travelling it does not find anything related to vacation, I assume it's because I'm not explicitly putting travelling in the synonyms file. Is there a way to activate stemming for all of the synonym terms in the file without having to manually put 'travel' and 'travelling' and 'travelers' in the synonym file? Thanks. -- View this message in context: http://www.nabble.com/stemming-the-synonyms-tp16945953p16945953.html Sent from the Solr - User mailing list archive at Nabble.com.
