I'm using the solr-nightly of 2008-04-05
Grant Ingersoll-6 wrote: > > What version of Solr/Lucene are you using? > > On Jul 28, 2008, at 4:53 PM, Britske wrote: > >> >> I'm on a development box currently and production servers will be >> bigger, but >> at the same time the index will be too. >> >> Each query requests at most 20 stored fields. Why doesn't help >> lazyfieldloading in this situation? >> I don't need to retrieve all stored fields and I thought I wasn't >> doing this >> (through limiting the fields returned using the FL-param), but if I >> read >> your comment correctly, apparently I am retrieving them all, I'm >> just not >> displaying them all? >> >> Also, if I understand correctly, for optimal performance I need to >> have at >> least enough RAM to put the entire Index size in OS cache (thus RAM) >> + the >> amount of RAM that SOLR / Lucene consumes directly through the JVM? >> (which >> among other things includes the Lucene field-cache + all of SOlr's >> caches on >> top of that). >> >> I've never read the requirement of having the entire index in OS cache >> before, is this because in normal situations (with less stored >> fields) it >> doesn't matter much? I'm just surprised to hear of this for the >> first time, >> since it will likely give a big impact on my design. >> >> Luckily most of the normal queries return 10 documents each, which >> results >> in a discrepancy between total elapsed time and qTIme of about 15-30 >> ms. >> Doesn't this seem strange, since to me it would seem logical that the >> discrepancy would be at least 1/10th of fetching 100 documents. >> >> hmm, hope you can shine some light on this, >> >> Thanks a lot, >> Britske >> >> >> >> Yonik Seeley wrote: >>> >>> That's a bit too tight to have *all* of the index cached...your best >>> bet is to go to 4GB+, or figure out a way not to have to retrieve so >>> many stored fields. >>> >>> -Yonik >>> >>> On Mon, Jul 28, 2008 at 4:27 PM, Britske <[EMAIL PROTECTED]> wrote: >>>> >>>> Size on disk is 1.84 GB (of which 1.3 GB sits in FDT files if that >>>> matters) >>>> Physical RAM is 2 GB with -Xmx800M set to Solr. >>>> >>>> >>>> Yonik Seeley wrote: >>>>> >>>>> That high of a difference is due to the part of the index >>>>> containing >>>>> these particular stored fields not being in OS cache. What's the >>>>> size >>>>> on disk of your index compared to your physical RAM? >>>>> >>>>> -Yonik >>>>> >>>>> On Mon, Jul 28, 2008 at 4:10 PM, Britske <[EMAIL PROTECTED]> wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> For some queries I need to return a lot of rows at once (say 100). >>>>>> When performing these queries I notice a big difference between >>>>>> qTime >>>>>> (which >>>>>> is mostly in the 15-30 ms range due to caching) and total time >>>>>> taken to >>>>>> return the response (measured through SolrJ's elapsedTime), >>>>>> which takes >>>>>> between 500-1600 ms. >>>>>> >>>>>> For queries which return less rows the difference becomes less >>>>>> big. >>>>>> >>>>>> I presume (after reading some threads in the past) that this is >>>>>> due to >>>>>> solr >>>>>> constructing and streaming the response (which includes >>>>>> retrieving the >>>>>> stored fields) , which is something that is not calculated in >>>>>> qTime. >>>>>> >>>>>> Documents have a lot of stored fields (more than 10.000), but at >>>>>> any >>>>>> given >>>>>> query a maximum of say 20 are returned (through fl-field ) or >>>>>> used (as >>>>>> part >>>>>> of filtering, faceting, sorting) >>>>>> >>>>>> I would have thought that enabling enableLazyFieldLoading for this >>>>>> situation >>>>>> would mean a lot, since so many stored fields can be skipped, >>>>>> but I >>>>>> notice >>>>>> no real difference in measuring total elapsed time (or qTime for >>>>>> that >>>>>> matter). >>>>>> >>>>>> Am I missing something here? What criteria would need to be met >>>>>> for a >>>>>> field >>>>>> to not be loaded for instance? Should I see a big performance >>>>>> boost in >>>>>> this >>>>>> situation? >>>>>> >>>>>> Thanks, >>>>>> Britske >>>>>> -- >>>>>> View this message in context: >>>>>> http://www.nabble.com/big-discrepancy-between-elapsedtime-and-qtime-although-enableLazyFieldLoading%3D-true-tp18698590p18698590.html >>>>>> Sent from the Solr - User mailing list archive at Nabble.com. >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> -- >>>> View this message in context: >>>> http://www.nabble.com/big-discrepancy-between-elapsedtime-and-qtime-although-enableLazyFieldLoading%3D-true-tp18698590p18698909.html >>>> Sent from the Solr - User mailing list archive at Nabble.com. >>>> >>>> >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/big-discrepancy-between-elapsedtime-and-qtime-although-enableLazyFieldLoading%3D-true-tp18698590p18699550.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> > > -------------------------- > Grant Ingersoll > http://www.lucidimagination.com > > Lucene Helpful Hints: > http://wiki.apache.org/lucene-java/BasicsOfPerformance > http://wiki.apache.org/lucene-java/LuceneFAQ > > > > > > > > > -- View this message in context: http://www.nabble.com/big-discrepancy-between-elapsedtime-and-qtime-although-enableLazyFieldLoading%3D-true-tp18698590p18699992.html Sent from the Solr - User mailing list archive at Nabble.com.
