It's metadata for a collection of 45 million documents that is mostly
photos, with some videos and text. The data is imported from a MySQL
database and split among six large shards (each nearly 13GB) and a small
shard with data added in the last week, which usually works out to
between 300,000 and 500,000 documents.
My goal is to reduce the index size without reducing the functionality.
Using copyField would just make it larger.
The biggest issue to solve is making sure that I don't have two terms
when there's a punctuation character at the beginning or end of a word.
For intstance, one chunk of text that I just analyzed ends up with
terms like the following, which are unneeded duplicates:
championship.
championship
'04
04
wisconsin.
wisconsin
Since I was already toying around, I just tested the whole notion with
the analysis tool. I configured two filter steps - the first with just
generateWordParts and catenateWords enabled, the second with all the
options including preserveOriginal enabled. A test analysis of input
with 59 whitespace separated words showed 93 terms with the single
filter and 77 with two. The only drop in term quality that I noticed
was that possessive words (apostrophe-s) no longer have the original
preserved. I haven't yet decided whether that's a problem.
Shawn
On 8/27/2010 11:00 AM, Erick Erickson wrote:
I agree with Marcus, the usefulness of passing through WDF twice
is suspect. You can always do a copyfield to a completely different
field and do whatever you want there, copyfield forks the raw input
to the second field, not the analyzed stream...
What is it you're really trying to accomplish? Your use-case would
help us help you.
About defining things differently in index and analysis. Sure, it can
make sense. But, especially with WDF it's tricky. Spend some
significant time in the admin analysis page looking at the effects
of various configurations before you decide.
Best
Erick
