+1 on Sujit's recommendation: we have a similar use case (detecting drug names / disease entities /MeSH terms ) and have been using the SolrTextTagger with great success.
We run a separate Solr instance as a tagging service and add the detected tags as metadata fields to a document before it is ingested into our main Solr collection. How many documents/product leaflets do you have ? The tagger is very fast at the Solr level but I'm seeing quite a bit of HTTP overhead. best -Simon On Fri, Sep 11, 2015 at 1:39 PM, Sujit Pal <sujit....@comcast.net> wrote: > Hi Francisco, > > >> I have many drug products leaflets, each corresponding to 1 product. In > the > other hand we have a medical dictionary with about 10^5 terms. > I want to detect all the occurrences of those terms for any leaflet > document. > Take a look at SolrTextTagger for this use case. > https://github.com/OpenSextant/SolrTextTagger > > 10^5 entries are not that large, I am using it for much larger dictionaries > at the moment with very good results. > > Its a project built (at least originally) by David Smiley, who is also > quite active in this group. > > -sujit > > > On Fri, Sep 11, 2015 at 7:29 AM, Alexandre Rafalovitch <arafa...@gmail.com > > > wrote: > > > Assuming the medical dictionary is constant, I would do a copyField of > > text into a separate field and have that separate field use: > > > > > http://www.solr-start.com/javadoc/solr-lucene/org/apache/lucene/analysis/miscellaneous/KeepWordFilterFactory.html > > with words coming from the dictionary (normalized). > > > > That way that new field will ONLY have your dictionary terms from the > > text. Then you can do facet against that field or anything else. Or > > even search and just be a lot more efficient. > > > > The main issue would be a gigantic filter, which may mean speed and/or > > memory issues. Solr has some ways to deal with such large set matches > > by compiling them into a state machine (used for auto-complete), but I > > don't know if that's exposed for your purpose. > > > > But could make a fun custom filter to build. > > > > Regards, > > Alex. > > ---- > > Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter: > > http://www.solr-start.com/ > > > > > > On 10 September 2015 at 22:21, Francisco Andrés Fernández > > <fra...@gmail.com> wrote: > > > Yes. > > > I have many drug products leaflets, each corresponding to 1 product. In > > the > > > other hand we have a medical dictionary with about 10^5 terms. > > > I want to detect all the occurrences of those terms for any leaflet > > > document. > > > Could you give me a clue about how is the best way to perform it? > > > Perhaps, the best way is (as Walter suggests) to do all the queries > every > > > time, as needed. > > > Regards, > > > > > > Francisco > > > > > > El jue., 10 de sept. de 2015 a la(s) 11:14 a. m., Alexandre > Rafalovitch < > > > arafa...@gmail.com> escribió: > > > > > >> Can you tell us a bit more about the business case? Not the current > > >> technical one. Because it is entirely possible Solr can solve the > > >> higher level problem out of the box without you doing manual term > > >> comparisons.In which case, your problem scope is not quite right. > > >> > > >> Regards, > > >> Alex. > > >> ---- > > >> Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter: > > >> http://www.solr-start.com/ > > >> > > >> > > >> On 10 September 2015 at 09:58, Francisco Andrés Fernández > > >> <fra...@gmail.com> wrote: > > >> > Hi all, I'm new to Solr. > > >> > I want to detect all ocurrences of terms existing in a thesaurus > into > > 1 > > >> or > > >> > more documents. > > >> > What´s the best strategy to make it? > > >> > Doing a query for each term doesn't seem to be the best way. > > >> > Many thanks, > > >> > > > >> > Francisco > > >> > > >
