So if you have a field that IS tokenized, regardless of what it's
called, then when you send "My Great Restaurant" to it for _indexing_,
it gets _tokenized upon indexing_ to seperate tokens: "My", "Great",
"Restaurant". Depending on what other analysis you have, it may get
further analyzed, perhaps to: "my", "great", "restaurant".
You don't need to seperate into tokens yourself before sending it to
Solr for indexing, if you define the field using a tokenizer, Solr will
do that when you index. Because this is a VERY common thing to do with
Solr; pretty much any field that you want to be effectively searchable
you have Solr tokenize like this.
Because Solr pretty much always matches on individual tokens, that's the
fundamental way Solr works.
Those seperate tokens is what allows you to SEARCH on the field, and get
a match on "my" or on "restaurant". If the field were non-tokenized,
you'd ONLY get a hit if the user entered "My Great Restaurant" (and
really not even then unless you take other actions, because of the way
Solr query parsers work you'll have trouble getting ANY hits to a
user-entered search with the 'lucene' or 'dismax' query parsers if you
don't tokenize).
That tokenized filed won't facet very well though -- if you facetted on
a tokenized field with that example entered in it, you'll get a facet
"my" with that item in it, and another facet "great" with that item in
it, and another facet "restuarant" with that item in it.
Which is why you likely want to use a seperate _untokenized_ field for
facetting. Which is why you end up wanting/needing two seperate fields
-- one that is tokenized for searching, and one that is not tokenized
(and usually not analyzed at all) for facetting.
Hope this helps.
On 5/2/2011 2:43 AM, elisabeth benoit wrote:
I'm a bit confused here.
What is the difference between CATEGORY and CATEGORY_TOKENIZED if I just do
a copyField from what field to another? And how can I search only for
Restaurant (fq= CATEGORY_TOKENIZED: Restaurant). Shouldn't I have something
like
<field name="CATEGORY_TOKENIZED">Hotel</field>, if I want this to work. And
from what I understand, this means I should do more then just copy
<field name="*CATEGORY*">Restaurant Hotel</field>
to CATEGORY_TOKENIZED.
Thanks,
Elisabeth
2011/4/28 Erick Erickson<erickerick...@gmail.com>
See below:
On Thu, Apr 28, 2011 at 9:03 AM, elisabeth benoit
<elisaelisael...@gmail.com> wrote:
yes, the multivalued field is not broken up into tokens.
so, if I understand well what you mean, I could have
a field CATEGORY with multiValued="true"
a field CATEGORY_TOKENIZED with multiValued=" true"
and then some POI
<field name="NAME">POI_Name</field>
...
<field name="*CATEGORY*">Restaurant Hotel</field>
<field name="CATEGORY_TOKENIZED">Restaurant</field>
<field name="CATEGORY_TOKENIZED">Hotel</field>
[EOE] If the above is the document you're sending, then no. The
document would be indexed with
<field name="*CATEGORY*">Restaurant Hotel</field>
<field name="CATEGORY_TOKENIZED">Restaurant Hotel</field>
Or even just:
<field name="*CATEGORY*">Restaurant Hotel</field>
and set up a<copyField> to copy the value from CATEGORY to
CATEGORY_TOKENIZED.
The multiValued part comes from:
"And a single POIs might have different categories so your document could
have"
which would look like:
<field name="CATEGORY">Restaruant Hotel</field>
<field name="CATEGORY">Health Spa</field>
<field name="CATEGORY">Dance Hall</field>
and your document would be counted for each of those entries while searches
against CATEGORY_TOKENIZED would match things like "dance" "spa" etc.
But do notice that if you did NOT want searching for "restaurant hall"
(no quotes),
to match then you could do proximity searches for less than your
increment gap. e.g.
(this time with the quotes) would be "restaurant hall"~50, which would then
NOT match if your increment gap were 100.
Best
Erick
do faceting on CATEGORY and fq on CATEGORY_TOKENIZED.
But then, wouldn't it be possible to do faceting on CATEGORY_TOKENIZED?
Best regards
Elisabeth
2011/4/28 Erick Erickson<erickerick...@gmail.com>
So, I assume your CATEGORY field is multiValued but each value is not
broken up into tokens, right? If that's the case, would it work to have
a
second field CATEGORY_TOKENIZED and run your fq against that
field instead?
You could have this be a multiValued field with an increment gap if you
wanted
to prevent matches across separate entries and have your fq do a
proximity
search where the proximity was less than the increment gap....
Best
Erick
On Thu, Apr 28, 2011 at 6:03 AM, elisabeth benoit
<elisaelisael...@gmail.com> wrote:
Hi Stefan,
Thanks for answering.
In more details, my problem is the following. I'm working on searching
points of interest (POIs), which can be hotels, restaurants, plumbers,
psychologists, etc.
Those POIs can be identified among other things by categories or by
brand.
And a single POIs might have different categories (no maximum number).
User
might enter a query like
McDonald’s Paris
or
Restaurant Paris
or
many other possible queries
First I want to do a facet search on brand and categories, to find out
which
case is the current case.
http://localhost:8080/solr /select?q=restaurant paris
&facet=true&facet.field=BRAND& facet.field=CATEGORY
and get an answer like
<lst name="facet_fields">
<lst name="CATEGORY">
<int name="Restaurant">598</int>
<int name="Restaurant Hotel">451</int>
Then I want to send a request with fq= CATEGORY: Restaurant and still
get
answers with CATEGORY= Restaurant Hotel.
One solution would be to modify the data to add a new document every
time
we
have a new category, so a POI with three different categories would be
index
three times, each time with a different category.
But I was wondering if there was another way around.
Thanks again,
Elisabeth
2011/4/28 Stefan Matheis<matheis.ste...@googlemail.com>
Hi Elisabeth,
that's not what FilterQueries are made for :) What against using that
Criteria in the Query?
Perhaps you want to describe your UseCase and we'll see if there's
another way to solve it?
Regards
Stefan
On Thu, Apr 28, 2011 at 9:09 AM, elisabeth benoit
<elisaelisael...@gmail.com> wrote:
Hello,
I would like to know if there is a way to use the fq parameter with
a
partial value.
For instance, if I have a request with fq=NAME:Joe, and I would
like
to
retrieve all answers where NAME contains Joe, including those with
NAME =
Joe Smith.
Thanks,
Elisabeth
