: I'll look into this. Thanks for the concrete example as I don't even
: know which classes to start to look at to implement such a feature.
Either roman isn't understanding what you are aksing for, or i'm not --
but i don't think what roman described will work for you...
: > so if your query contains no duplicates and all terms must match, you can
: > be sure that you are collecting docs only when the number of terms matches
: > number of clauses in the query
several of the examples you gave did not match what Roman is describing,
as i understand it. Most people on this thread seem to be getting
confused by having their perceptions "flipped" about what your "data known
in advance is" vs the "data you get at request time".
You described this...
: >>>>> Product keyword: "Sony"
: >>>>> Product keyword: "Samsung Galaxy"
: >>>>>
: >>>>> We would like to be able to detect given a product title whether or
: >> not it
: >>>>> matches any known keywords. For a keyword to be matched all of it's
: >> terms
: >>>>> must be present in the product title given.
: >>>>>
: >>>>> Product Title: "Sony Experia"
: >>>>> Matches and returns a highlight: "<em>Sony</em> Experia"
...suggesting that what you call "product keywords" are the "data you know
about in advance" and "product titles" are the data you get at request
time.
So your example of the "request time" input (ie: query) "Sony Experia"
matching "data known in advance (ie: indexed document) "Sony" would not
work with Roman's example.
To rephrase (what i think i understand is) your goal...
* you have many (10*3+) documents known in advance
* any document D contain a set of words W(D) of varing sizes
* any requests Q contains a set of words W(Q) of varing izes
* you want a given request R to match a document D if and only if:
- W(D) is a subset of W(Q)
- ie: no iten exists in W(D) that does not exist in W(Q)
- ie: any number of items may exist in W(Q) that are not in W(D)
So to reiteratve your examples from before, but change the "labels" a
bit and add some more converse examples (and ignore the "highlighting"
aspect for a moment...
doc1 = "Sony"
doc2 = "Samsung Galaxy"
doc3 = "Sony Playstation"
queryA = "Sony Experia" ... matches only doc1
queryB = "Sony Playstation 3" ... matches doc3 and doc1
queryC = "Samsung 52inch LC" ... doesn't match anything
queryD = "Samsung Galaxy S4" ... matches doc2
queryE = "Galaxy Samsung S4" ... matches doc2
...do i still have that correct?
A similar question came up in the past, but i can't find my response now
so i'll try to recreate it ...
1) if you don't care about using non-trivial analysis (ie: you don't need
stemming, or synonyms, etc..), you can do this with some
really simple function queries -- asusming you index a field containing
hte number of "words" in each document, in addition to the words
themselves. Assuming your words are in a field named "words" and the
number of words is in a field named "words_count" a request for something
like "Galaxy Samsung S4" can be represented as...
q={!frange l=0 u=0}sub(words_count,
sum(termfreq('words','Galaxy'),
termfreq('words','Samsung'),
termfreq('words','S4'))
...ie: you want to compute the sub of the term frequencies for each of
hte words requested, and then you want ot subtract that sum from the
number of terms in the documengt -- and then you only want ot match
documents where the result of that subtraction is 0.
one complexity that comes up, is that you haven't specified:
* can the list of words in your documents contain duplicates?
* can the list of words in your query contain duplicates?
* should a document with duplicatewords match only if the query also
contains the same word duplicated?
...the answers to those questions make hte math more complicated (and are
left as an excersize for the reader)
2) if you *do* care about using non-trivial analysis, then you can't use
the simple "termfreq()" function, which deals with raw terms -- in stead
you have to use the "query()" function to ensure that the input is parsed
appropriately -- but then you have to wrap that function in something that
will normalize the scores - so in place of termfreq('words','Galaxy')
you'd want something like...
if(query({!field f=words v='Galaxy'}),1,0)
...but again the math gets much harder if you make things more complex
with duplicate words i nthe document or duplicate words in the query -- you'd
probably have to use a custom similarity to get the scores returned by the
query() function to be usable as is in the match equation (and drop the
"if()" function)
As for the highlighting part of hte problme -- that becomes much easier --
independent of the queries you use to *match* the documents, you can then
specify a "hl.q" param to specify a much simpler query just containing the
basic lst of words (as a simple boolean query, all clouses optional) and
let it highlight them in your list of words.
-Hoss
