What version of Solr/Lucene are you using?
On Jul 28, 2008, at 4:53 PM, Britske wrote:
I'm on a development box currently and production servers will be
bigger, but
at the same time the index will be too.
Each query requests at most 20 stored fields. Why doesn't help
lazyfieldloading in this situation?
I don't need to retrieve all stored fields and I thought I wasn't
doing this
(through limiting the fields returned using the FL-param), but if I
read
your comment correctly, apparently I am retrieving them all, I'm
just not
displaying them all?
Also, if I understand correctly, for optimal performance I need to
have at
least enough RAM to put the entire Index size in OS cache (thus RAM)
+ the
amount of RAM that SOLR / Lucene consumes directly through the JVM?
(which
among other things includes the Lucene field-cache + all of SOlr's
caches on
top of that).
I've never read the requirement of having the entire index in OS cache
before, is this because in normal situations (with less stored
fields) it
doesn't matter much? I'm just surprised to hear of this for the
first time,
since it will likely give a big impact on my design.
Luckily most of the normal queries return 10 documents each, which
results
in a discrepancy between total elapsed time and qTIme of about 15-30
ms.
Doesn't this seem strange, since to me it would seem logical that the
discrepancy would be at least 1/10th of fetching 100 documents.
hmm, hope you can shine some light on this,
Thanks a lot,
Britske
Yonik Seeley wrote:
That's a bit too tight to have *all* of the index cached...your best
bet is to go to 4GB+, or figure out a way not to have to retrieve so
many stored fields.
-Yonik
On Mon, Jul 28, 2008 at 4:27 PM, Britske <[EMAIL PROTECTED]> wrote:
Size on disk is 1.84 GB (of which 1.3 GB sits in FDT files if that
matters)
Physical RAM is 2 GB with -Xmx800M set to Solr.
Yonik Seeley wrote:
That high of a difference is due to the part of the index
containing
these particular stored fields not being in OS cache. What's the
size
on disk of your index compared to your physical RAM?
-Yonik
On Mon, Jul 28, 2008 at 4:10 PM, Britske <[EMAIL PROTECTED]> wrote:
Hi all,
For some queries I need to return a lot of rows at once (say 100).
When performing these queries I notice a big difference between
qTime
(which
is mostly in the 15-30 ms range due to caching) and total time
taken to
return the response (measured through SolrJ's elapsedTime),
which takes
between 500-1600 ms.
For queries which return less rows the difference becomes less
big.
I presume (after reading some threads in the past) that this is
due to
solr
constructing and streaming the response (which includes
retrieving the
stored fields) , which is something that is not calculated in
qTime.
Documents have a lot of stored fields (more than 10.000), but at
any
given
query a maximum of say 20 are returned (through fl-field ) or
used (as
part
of filtering, faceting, sorting)
I would have thought that enabling enableLazyFieldLoading for this
situation
would mean a lot, since so many stored fields can be skipped,
but I
notice
no real difference in measuring total elapsed time (or qTime for
that
matter).
Am I missing something here? What criteria would need to be met
for a
field
to not be loaded for instance? Should I see a big performance
boost in
this
situation?
Thanks,
Britske
--
View this message in context:
http://www.nabble.com/big-discrepancy-between-elapsedtime-and-qtime-although-enableLazyFieldLoading%3D-true-tp18698590p18698590.html
Sent from the Solr - User mailing list archive at Nabble.com.
--
View this message in context:
http://www.nabble.com/big-discrepancy-between-elapsedtime-and-qtime-although-enableLazyFieldLoading%3D-true-tp18698590p18698909.html
Sent from the Solr - User mailing list archive at Nabble.com.
--
View this message in context:
http://www.nabble.com/big-discrepancy-between-elapsedtime-and-qtime-although-enableLazyFieldLoading%3D-true-tp18698590p18699550.html
Sent from the Solr - User mailing list archive at Nabble.com.
--------------------------
Grant Ingersoll
http://www.lucidimagination.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
