I have noticed a significant difference in filter cache warming times on
my shards between 3.2 and 1.4.1. What can I do to troubleshoot this?
Please let me know what additional information you might need to look
deeper. I know this isn't enough.
It takes about 3 seconds to do an autowarm count of 8 on 1.4.1 and 10-15
seconds to do an autowarm count of 4 on 3.2. The only explicit warming
query is *:*, sorted descending by post_date, a tlong field containing a
UNIX timestamp, precisionStep 16. The indexes are not entirely
identical, but the new one did evolve from the old one. Perhaps one of
the experts might spot something that makes for much slower filter cache
warming, or some way to look deeper if this seems wrong? Is there a way
to see the search URL bits that populated the cache?
Index differences: The new index has four extra small fields, is no
longer removing stopwords, and has omitTermFreqAndPositions enabled on a
significant number of fields. Most of the fields are tokenized text,
and now more than half of those don't have tf and tp enabled. Naturally
the largest text field where most of the matches happen still does have
them enabled.
To increase reindex speed, the new index has a termIndexInterval of
1024, the old one is at the default of 128. In terms of raw size, the
new index is less than one percent larger than the old one. The old
shards average out to 17.22GB, the new ones to 17.41GB. Here's an
overview of the differences of each type of file (comparing the huge
optimized segment only, not the handful of tiny ones since) on one the
index with the largest size gap, old value listed first:
fdt: 6317180127/6055634923 (4.1% decrease)
fdx: 76447972/75647412 (1% decrease)
fnm: 382, 338 (44 bytes! woohoo!)
frq: 2828400926/2873249038 (1.5% increase)
nrm: 28367782/38223988 (35% increase)
prx: 2449154203/2684249069 (9.5% increase)
tii: 1686298/13329832 (790% increase) ****
tis: 923045932/999294109 (8% increase)
tvd: 18910972/19111840 (1% increase)
tvf: 5867309063/5640332282 (3.9% decrease)
tvx: 151294820/152895940 (1% increase)
The tii and nrm files are the only ones that saw a significant size
increase, but the tii file is MUCH bigger.
Thanks,
Shawn
