Alex, I added your comments to SOLR-3758 (https://issues.apache.org/jira/browse/SOLR-3758) , which seems to me to be the very same issue.
If you need this to work now and if you cannot devise a fix yourself, then perhaps a workaround is if the query returns with 0 results, re-issue the query with "&rows=0&group=false" (you would omit all other optional components also). This will give you back just a spell check result. I realize this is not optimal because it requires the overhead of issuing 2 queries but if you do it only in instances the user gets nothing (or very little) back maybe it would be tolerable? Then once a viable fix is devised you can remove the extra code from your application. James Dyer Ingram Content Group (615) 213-4311 -----Original Message----- From: alx...@aim.com [mailto:alx...@aim.com] Sent: Friday, March 22, 2013 12:53 PM To: solr-user@lucene.apache.org Subject: Re: strange behaviour of wordbreak spellchecker in solr cloud Hello, Further investigation shows the following pattern, for both DirectIndex and wordbreak spellchekers. Assume that in all cases there are spellchecker results when distrib=false In distributed mode (distrib=true) case when matches=0 1. group=true, no spellcheck results 2. group=false , there are spellcheck results case when matches>0 1. group=true, there are spellcheck results 2. group =false, there are spellcheck results Do these constitute a failing test case? Thanks. Alex. -----Original Message----- From: alxsss <alx...@aim.com> To: solr-user <solr-user@lucene.apache.org> Sent: Thu, Mar 21, 2013 6:50 pm Subject: Re: strange behaviour of wordbreak spellchecker in solr cloud Hello, I am debugging the SpellCheckComponent#finishStage. >From the responses I see that not only wordbreak, but also directSpellchecker does not return some results in distributed mode. The request handler I was using had <str name="group">true</str> So, I desided to turn of grouping and I see spellcheck results in distributed mode. curl 'server1:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10&shards.qt=testhandler' has no spellchek results but curl 'server1:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10&shards.qt=testhandler &group=false' returns results. So, the conclusion is that grouping causes the distributed spellcheker to fail. Could please you point me to the class that may be responsible to this issue? Thanks. Alex. -----Original Message----- From: Dyer, James <james.d...@ingramcontent.com> To: solr-user <solr-user@lucene.apache.org> Sent: Thu, Mar 21, 2013 11:23 am Subject: RE: strange behaviour of wordbreak spellchecker in solr cloud The shard responses get combined in SpellCheckComponent#finishStage . I highly recommend you file a JIRA bug report for this at https://issues.apache.org/jira/browse/SOLR . If you write a failing unit test, it would make it much more likely that others would help you with a fix. Of course, if you solve the issue entirely, a patch would be much appreciated. James Dyer Ingram Content Group (615) 213-4311 -----Original Message----- From: alx...@aim.com [mailto:alx...@aim.com] Sent: Thursday, March 21, 2013 12:45 PM To: solr-user@lucene.apache.org Subject: Re: strange behaviour of wordbreak spellchecker in solr cloud Hello, We need this feature be fixed ASAP. So, please let me know which class is responsible for combining spellcheck results from all shards. I will try to debug the code. Thanks in advance. Alex. -----Original Message----- From: alxsss <alx...@aim.com> To: solr-user <solr-user@lucene.apache.org> Sent: Tue, Mar 19, 2013 11:34 am Subject: Re: strange behaviour of wordbreak spellchecker in solr cloud -- distributed environment. But to nail it down, we probably need to see both -- the applicable <requestHandler /> Not sure what this is? I have <searchComponent name="spellcheck" class="solr.SpellCheckComponent"> <str name="queryAnalyzerFieldType">spell</str> <!-- Multiple "Spell Checkers" can be declared and used by this component --> <!-- a spellchecker built from a field of the main index --> <lst name="spellchecker"> <str name="name">direct</str> <str name="field">spell</str> <str name="classname">solr.DirectSolrSpellChecker</str> <!-- the spellcheck distance measure used, the default is the internal levenshtein --> <str name="distanceMeasure">internal</str> <!-- minimum accuracy needed to be considered a valid spellcheck suggestion --> <float name="accuracy">0.5</float> <!-- the maximum #edits we consider when enumerating terms: can be 1 or 2 --> <int name="maxEdits">2</int> <!-- the minimum shared prefix when enumerating terms --> <int name="minPrefix">1</int> <!-- maximum number of inspections per result. --> <int name="maxInspections">5</int> <!-- minimum length of a query term to be considered for correction --> <int name="minQueryLength">4</int> <!-- maximum threshold of documents a query term can appear to be considered for correction --> <float name="maxQueryFrequency">0.01</float> <!-- uncomment this to require suggestions to occur in 1% of the documents <float name="thresholdTokenFrequency">.01</float> --> </lst> <!-- a spellchecker that can break or combine words. See "/spell" handler below for usage --> <lst name="spellchecker"> <str name="name">wordbreak</str> <str name="classname">solr.WordBreakSolrSpellChecker</str> <str name="field">spell</str> <str name="combineWords">true</str> <str name="breakWords">true</str> <int name="maxChanges">10</int> </lst> <!-- a spellchecker that uses a different distance measure --> <!-- <lst name="spellchecker"> <str name="name">jarowinkler</str> <str name="field">spell</str> <str name="classname">solr.DirectSolrSpellChecker</str> <str name="distanceMeasure"> org.apache.lucene.search.spell.JaroWinklerDistance </str> </lst> --> <!-- a spellchecker that use an alternate comparator comparatorClass be one of: 1. score (default) 2. freq (Frequency first, then score) 3. A fully qualified class name --> <!-- <lst name="spellchecker"> <str name="name">freq</str> <str name="field">lowerfilt</str> <str name="classname">solr.DirectSolrSpellChecker</str> <str name="comparatorClass">freq</str> --> <!-- A spellchecker that reads the list of words from a file --> <!-- <lst name="spellchecker"> <str name="classname">solr.FileBasedSpellChecker</str> <str name="name">file</str> <str name="sourceLocation">spellings.txt</str> <str name="characterEncoding">UTF-8</str> <str name="spellcheckIndexDir">spellcheckerFile</str> </lst> --> </searchComponent> spell filed in our schema is called spell and its type also is called spell. Here are requests curl 'server1:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10&shards.qt=testhandler&distrib=false' <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">32</int> <lst name="params"> <str name="indent">true</str> <str name="shards.qt">testhandler</str> <str name="q">paulusoles</str> <str name="distrib">false</str> <str name="rows">10</str> </lst> </lst> <lst name="grouped"> <lst name="site"> <int name="matches">0</int> <int name="ngroups">0</int> <arr name="groups"/> </lst> </lst> <lst name="highlighting"/> <lst name="spellcheck"> <lst name="suggestions"/> </lst> </response> curl 'server2:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10&shards.qt=testhandler&distrib=false' <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">26</int> <lst name="params"> <str name="indent">true</str> <str name="shards.qt">testhandler</str> <str name="q">paulusoles</str> <str name="distrib">false</str> <str name="rows">10</str> </lst> </lst> <lst name="grouped"> <lst name="site"> <int name="matches">0</int> <int name="ngroups">0</int> <arr name="groups"/> </lst> </lst> <lst name="highlighting"/> <lst name="spellcheck"> <lst name="suggestions"> <lst name="paulusoles"> <int name="numFound">1</int> <int name="startOffset">0</int> <int name="endOffset">11</int> <arr name="suggestion"> <str>paul u soles</str> </arr> </lst> <str name="collation">(paul u soles)</str> </lst> </lst> </response> No distrib param curl 'server1:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10&shards.qt=testhandler' <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">24</int> <lst name="params"> <str name="indent">true</str> <str name="shards.qt">testhandler</str> <str name="q">paulusoles</str> <str name="distrib">false</str> <str name="rows">10</str> </lst> </lst> <lst name="grouped"> <lst name="site"> <int name="matches">0</int> <int name="ngroups">0</int> <arr name="groups"/> </lst> </lst> <lst name="highlighting"/> <lst name="spellcheck"> <lst name="suggestions"/> </lst> </response> curl 'server2:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10&shards.qt=testhandler' <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">24</int> <lst name="params"> <str name="indent">true</str> <str name="shards.qt">testhandler</str> <str name="q">paulusoles</str> <str name="rows">10</str> </lst> </lst> <lst name="grouped"> <lst name="site"> <int name="matches">0</int> <int name="ngroups">0</int> <arr name="groups"/> </lst> </lst> <lst name="highlighting"/> <lst name="spellcheck"> <lst name="suggestions"/> </lst> </response> Thanks. Alex. ---Original Message----- From: Dyer, James <james.d...@ingramcontent.com> To: solr-user <solr-user@lucene.apache.org> Sent: Tue, Mar 19, 2013 11:10 am Subject: RE: strange behaviour of wordbreak spellchecker in solr cloud You may likely be hitting on a bug with WordBreakSolrSpellChecker in a distributed environment. But to nail it down, we probably need to see both the applicable <requestHandler /> section of your config and also this section: <searchComponent name="spellcheck" class="solr.SpellCheckComponent" />. Also need an example of a query that succeeds non-distributed (with the exact query url and output you get) vs the same query url and output in the distributed scenario. Then, without access to your actual index, it might be possible to come up with a failing unit test. With a failing unit test in hand, we have a good shot at getting a fix. James Dyer Ingram Content Group (615) 213-4311 -----Original Message----- From: alx...@aim.com [mailto:alx...@aim.com] Sent: Tuesday, March 19, 2013 12:39 PM To: solr-user@lucene.apache.org Subject: Re: strange behaviour of wordbreak spellchecker in solr cloud Hello, I was testing my custom testhandler. Direct spellchecker also was not working in cloud. After I added <arr name="last-components"> <str>spellcheck</str> </arr> to /select requestHandler it worked but the wordbreak spellchecker. I have added shards.qt=testhanlder to curl request but it did not solve the issue. Thanks. Alex. -----Original Message----- From: Dyer, James <james.d...@ingramcontent.com> To: solr-user <solr-user@lucene.apache.org> Sent: Tue, Mar 19, 2013 10:30 am Subject: RE: strange behaviour of wordbreak spellchecker in solr cloud Mark, I wasn't sure if Alex is actually testing /select, or if the problem is just coming up in /testhandler. Just wanted to verify that before we get into bug reports. DistributedSpellCheckComponentTest does have 1 little Word Break test scenario in it, so we know WordBreakSolrSpellChecker at least works some of the time in a Distributed environment :) . Ideally, we should probably use a random test for stuff like this as adding a bunch of test scenarios would make this already-slower-than-molasses test even slower. On the other hand, we want to test as many possibilities as we can. Based on DSCCT and it being so superficial, I really can't vouch too much for my spell check enhancements working as well with shards as they do with a single index. James Dyer Ingram Content Group (615) 213-4311 -----Original Message----- From: Mark Miller [mailto:markrmil...@gmail.com] Sent: Tuesday, March 19, 2013 11:49 AM To: solr-user@lucene.apache.org Subject: Re: strange behaviour of wordbreak spellchecker in solr cloud My first thought too, but then I saw that he had the spell component in both his custom testhander and the /select handler, so I'd expect that to work as well. - Mark On Mar 19, 2013, at 12:18 PM, "Dyer, James" <james.d...@ingramcontent.com> wrote: > Can you try including in your request the "shards.qt" parameter? In your case, I think you should set it to "testhandler". See http://wiki.apache.org/solr/SpellCheckComponent?highlight=%28shards\.qt%29#Distributed_Search_Support for a brief discussion. > > James Dyer > Ingram Content Group > (615) 213-4311 > > > -----Original Message----- > From: alx...@aim.com [mailto:alx...@aim.com] > Sent: Monday, March 18, 2013 4:07 PM > To: solr-user@lucene.apache.org > Subject: strange behaviour of wordbreak spellchecker in solr cloud > > Hello, > > I try to use wordbreak spellchecker in solr-4.2 with cloud feature. We have two server with one shard in each of them. > > curl 'server1:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10' > curl 'server2:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10' > > does not return any results in spellchecker. However, if I specify distrib=false only one of these has spellchecker results. > > curl > 'server1:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10&distrib=false' > > no spellcheler results > > curl > 'server2:8983/solr/test/testhandler?q=paulusoles&indent=true&rows=10&distrib=false' > returns spellcheker results. > > > My testhandler and select handlers are as follows > > > <requestHandler name="/testhandler" class="solr.SearchHandler" > > <lst name="defaults"> > <str name="defType">edismax</str> > <str name="echoParams">explicit</str> > <float name="tie">0.01</float> > <str name="qf">host^30 content^0.5 title^1.2 </str> > <str name="pf">site^25 content^10 title^22</str> > <str name="fl">url,id,title</str> > <!-- <str name="mm">2<-1 5<-3 6<90%</str> --> > <str name="mm">3<-1 5<-3 6<90%</str> > <int name="ps">1</int> > > <str name="hl">true</str> > <str name="hl.fl">content</str> > <str name="f.content.hl.fragmenter">regex</str> > <str name="hl.fragsize">165</str> > <str name="hl.fragmentsBuilder">default</str> > > > <str name="spellcheck.dictionary">direct</str> > <str name="spellcheck.dictionary">wordbreak</str> > <str name="spellcheck">on</str> > <str name="spellcheck.collate">true</str> > <str name="spellcheck.onlyMorePopular">false</str> > <str name="spellcheck.count">2</str> > > </lst> > > <arr name="last-components"> > <str>spellcheck</str> > </arr> > > </requestHandler> > > > <requestHandler name="/select" class="solr.SearchHandler"> > <!-- default values for query parameters can be specified, these > will be overridden by parameters in the request > --> > <lst name="defaults"> > <str name="echoParams">explicit</str> > <int name="rows">10</int> > <!-- <str name="df">text</str> --> > </lst> > <!-- In addition to defaults, "appends" params can be specified > to identify values which should be appended to the list of > multi-val params from the query (or the existing "defaults"). > --> > <!-- In this example, the param "fq=instock:true" would be appended to > any query time fq params the user may specify, as a mechanism for > partitioning the index, independent of any user selected filtering > that may also be desired (perhaps as a result of faceted searching). > > NOTE: there is *absolutely* nothing a client can do to prevent these > "appends" values from being used, so don't use this mechanism > unless you are sure you always want it. > --> > <!-- > <lst name="appends"> > <str name="fq">inStock:true</str> > </lst> > --> > <!-- "invariants" are a way of letting the Solr maintainer lock down > the options available to Solr clients. Any params values > specified here are used regardless of what values may be specified > in either the query, the "defaults", or the "appends" params. > > In this example, the facet.field and facet.query params would > be fixed, limiting the facets clients can use. Faceting is > not turned on by default - but if the client does specify > facet=true in the request, these are the only facets they > will be able to see counts for; regardless of what other > facet.field or facet.query params they may specify. > > NOTE: there is *absolutely* nothing a client can do to prevent these > "invariants" values from being used, so don't use this mechanism > unless you are sure you always want it. > --> > <!-- > <lst name="invariants"> > <str name="facet.field">cat</str> > <str name="facet.field">manu_exact</str> > <str name="facet.query">price:[* TO 500]</str> > <str name="facet.query">price:[500 TO *]</str> > </lst> > --> > <!-- If the default list of SearchComponents is not desired, that > list can either be overridden completely, or components can be > prepended or appended to the default list. (see below) > --> > <!-- > <arr name="components"> > <str>nameOfCustomComponent1</str> > <str>nameOfCustomComponent2</str> > </arr> > --> > <arr name="last-components"> > <str>spellcheck</str> > </arr> > </requestHandler> > > > > is this a bug or something else has to be done? > > > Thanks. > Alex. >
