Hi Grant, My queries are about 5 times slower when using payloads as compared to queries that dont use payloads on the same index. I have not done any profiling yet, I am trying out lucid gaze now. I do all the load testing after warming up. Since my index is small ~1 GB, was wondering if a ramDirectory will help instead of the default Directory implementation for the indexReader?
Thanks, Raghu On Thu, Dec 17, 2009 at 6:58 PM, Grant Ingersoll <gsing...@apache.org>wrote: > > On Dec 17, 2009, at 4:52 AM, Raghuveer Kancherla wrote: > > > Hi, > > With help from the group here, I have been able to set up a search > > application with payloads enabled. However, there is a noticeable > increase > > in query response times with payloads as compared to the same queries > > without payloads. I am also seeing a lot more disk IO (I have a 7200 rpm > > disk) and comparatively lesser cpu usage. > > > > I am guessing this is because of the use of payloadTermQuery and > > payloadNearQuery both of which extend SpanQuery formats. SpanQueries > read > > the positions index which will be much larger than the index accessed by > a > > simple TermQuery. > > > > Is there any way of making this system faster without having to > distribute > > the index. My index size is hardly 1GB (~200k documents and only one > field > > to search in). I am experiencing query times as high as 2 seconds > (average). > > > > Any indications on the direction in which I can experiment will also be > very > > helpful. > > > > Yeah, payloads are going to be slower, but how much slower are they for > you? Are you warming up those queries? > > Also, have you done any profiling? > > > > I looked at HathiTrust digital library articles. The methods indicated > there > > talk about avoiding reading the positions index (converting PhraseQueries > to > > TermQueries). That will not work in my case because, I still have to read > > the positions index to get the payload information during scoring. Let me > > know if my understanding is incorrect. > > > > > > Thanks, > > -Raghu > > -------------------------- > Grant Ingersoll > http://www.lucidimagination.com/ > > Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using > Solr/Lucene: > http://www.lucidimagination.com/search > >
