Shane: You've covered all the config stuff that I can think of. There's one other possibility. Do you have the soft commits turned on and are they very short? Although soft commits shouldn't invalidate any segment-level caches (but I'm not sure whether the sorting buffers are low-level or not).
About the only other thing I can think of is that you're somehow doing hard commits from, say, the client but that's really stretching. All I can really say at this point is that this isn't a problem I've seen before, so it's _likely_ some innocent-seeming config has changed. I'm sure it'll be obvious once you find it <G>... Erick On Wed, Jun 12, 2013 at 11:51 PM, Shane Perry <thry...@gmail.com> wrote: > Erick, > > I agree, it doesn't make sense. I manually merged the solrconfig.xml from > the distribution example with my 3.6 solrconfig.xml, pulling out what I > didn't need. There is the possibility I removed something I shouldn't have > though I don't know what it would be. Minus removing the dynamic fields, a > custom tokenizer class, and changing all my fields to be stored, the > schema.xml file should be the same as well. I'm not currently in the > position to do so, but I'll double check those two files. Finally, the > data was re-indexed when I moved to 4.3. > > My statement about field values wasn't stated very well. What I meant is > that the 'text' field has more unique terms than some of my other fields. > > As for this being an edge case, I'm not sure why it would manifest itself > in 4.3 but not in 3.6 (short of me having a screwy configuration setting). > If I get a chance, I'll see if I can duplicate the behavior with a small > document count in a sandboxed environment. > > Shane > > On Wed, Jun 12, 2013 at 5:14 PM, Erick Erickson > <erickerick...@gmail.com>wrote: > >> This doesn't make much sense, particularly the fact >> that you added first/new searchers. I'm assuming that >> these are sorting on the same field as your slow query. >> >> But sorting on a text field for which >> "Overall, the values of the field are unique" >> is a red-flag. Solr doesn't sort on fields that have >> more than one term, so you might as well use a >> string field and be done with it, it's possible you're >> hitting some edge case. >> >> Did you just copy your 3.6 schema and configs to >> 4.3? Did you re-index? >> >> Best >> Erick >> >> On Wed, Jun 12, 2013 at 5:11 PM, Shane Perry <thry...@gmail.com> wrote: >> > Thanks for the responses. >> > >> > Setting first/newSearcher had no noticeable effect. I'm sorting on a >> > stored/indexed field named 'text' who's fieldType is solr.TextField. >> > Overall, the values of the field are unique. The JVM is only using about >> > 2G of the available 12G, so no OOM/GC issue (at least on the surface). >> The >> > server is question is a slave with approximately 56 million documents. >> > Additionally, sorting on a field of the same type but with significantly >> > less uniqueness results quick response times. >> > >> > The following is a sample of *debugQuery=true* for a query which returns >> 1 >> > document: >> > >> > <lst name="process"> >> > <double name="time">61458.0</double> >> > <lst name="query"> >> > <double name="time">61452.0</double> >> > </lst> >> > <lst name="facet"> >> > <double name="time">0.0</double> >> > </lst> >> > <lst name="mlt"> >> > <double name="time">0.0</double> >> > </lst> >> > <lst name="highlight"> >> > <double name="time">0.0</double> >> > </lst> >> > <lst name="stats"> >> > <double name="time">0.0</double> >> > </lst> >> > <lst name="debug"> >> > <double name="time">6.0</double> >> > </lst> >> > </lst> >> > >> > >> > -- Update -- >> > >> > Out of desperation, I turned off replication by commenting out the *<list >> > name="slave">* element in the replication requestHandler block. After >> > restarting tomcat I was surprised to find that the replication admin UI >> > still reported the core as replicating. Search queries were still slow. >> I >> > then disabled replication via the UI and the display updated to report >> the >> > core was no longer replicating. Queries are now fast so it appears that >> > the sorting may be a red-herring. >> > >> > It's may be of note to also mention that the slow queries don't appear to >> > be getting cached. >> > >> > Thanks again for the feed back. >> > >> > On Wed, Jun 12, 2013 at 2:33 PM, Jack Krupansky <j...@basetechnology.com >> >wrote: >> > >> >> Rerun the sorted query with &debugQuery=true and look at the module >> >> timings. See what stands out >> >> >> >> Are you actually sorting on a "text" field, as opposed to a "string" >> field? >> >> >> >> Of course, it's always possible that maybe you're hitting some odd >> OOM/GC >> >> condition as a result of Solr growing between releases. >> >> >> >> -- Jack Krupansky >> >> >> >> -----Original Message----- From: Shane Perry >> >> Sent: Wednesday, June 12, 2013 3:00 PM >> >> To: solr-user@lucene.apache.org >> >> Subject: Sorting by field is slow >> >> >> >> >> >> In upgrading from Solr 3.6.1 to 4.3.0, our query response time has >> >> increased exponentially. After testing in 4.3.0 it appears the same >> query >> >> (with 1 matching document) returns after 100 ms without sorting but >> takes 1 >> >> minute when sorting by a text field. I've looked around but haven't yet >> >> found a reason for the degradation. Can someone give me some insight or >> >> point me in the right direction for resolving this? In most cases, I >> can >> >> change my code to do client-side sorting but I do have a couple of >> >> situations where pagination prevents client-side sorting. Any help >> would >> >> be greatly appreciated. >> >> >> >> Thanks, >> >> >> >> Shane >> >> >>
