Hmmm, we're not communicating <G>... The update processor wouldn't search in the classic sense. It would just use lower-level index traversal to determine if the doc (identified by your unique key) was already in the index and skip indexing that document if it was. No real *searching* involved (see TermDocs.seek for one approach).
The price would be that you are transmitting the document over to the Solr instance and then throwing it away. Best Erick On Thu, Dec 29, 2011 at 12:52 AM, Mikhail Khludnev <mkhlud...@griddynamics.com> wrote: > Alexander, > > I have two ideas how to implement fast dedupe externally, assuming your PKs > don't fit to java.util.*Map: > > - your crawler can use inprocess RDBMS (Derby, H2) to track dupes; > - if your crawler is stateless - it doesn't track PKs which has been > already crawled, you can retrieve it from Solr via > http://wiki.apache.org/solr/TermsComponent . That's blazingly fast, but > it might be a problem with removed documents (I'm not sure). And it's also > can lead to OOMException (if you have too much PKs). Let me know if you > need a workaround for one of these problems. > > If you choose internal dedupe (UpdateProcessor), pls let me know if > querying one-by-one will be to slow for your and you'll need to do it > page-by-page. I did some of such paging, and will do something similar > soon, so I'm interested in it. > > Regards > > On Thu, Dec 29, 2011 at 9:34 AM, Alexander Aristov < > alexander.aris...@gmail.com> wrote: > >> Unfortunately I have a lot of duplicates and taking that searching might >> suffer I will try with implementing update procesor. >> >> But your idea is interesting and I will consider it, thanks. >> >> Best Regards >> Alexander Aristov >> >> >> On 28 December 2011 19:12, Tanguy Moal <tanguy.m...@gmail.com> wrote: >> >> > Hello Alexander, >> > >> > I don't know much about your requirements in terms of size and >> > performances, but I've had a similar use case and found a pretty simple >> > workaround. >> > If your duplicate rate is not too high, you can have the >> > SignatureProcessor to generate fingerprint of documents (you already did >> > that). >> > >> > Simply turn off overwritting of duplicates, you can then rely on solr's >> > grouping / field collapsing to group your search results by fingerprints. >> > You'll then have one document group per "real" document. You can use >> > group.sort to sort your groups by indexing date ascending, and >> > group.limit=1 to keep only the oldest one. >> > You can even use group.format = simple to serve results as if no >> > collapsing occured, and use group.ngroups (/!\ could be expansive /!\) to >> > get the real number of deduplicated documents. >> > >> > Of course the index will be larger, as I said, I made no assumption >> > regarding your operating requirements. And search can be a bit slower, >> > depending on the average rate of duplicated documents. >> > But you've got your issue addressed by configuration tuning only... >> > Depending on your project's sizing, it could be time saving. >> > >> > The advantage is that you have the precious information of what content >> is >> > duplicated from where :-) >> > >> > Hope this helps, >> > >> > -- >> > Tanguy >> > >> > Le 28/12/2011 15:45, Alexander Aristov a écrit : >> > >> > Thanks Eric, >> >> >> >> it sets me direction. I will be writing new plugin and will get back to >> >> the >> >> dev forum with results and then we will decide next steps. >> >> >> >> Best Regards >> >> Alexander Aristov >> >> >> >> >> >> On 28 December 2011 18:08, Erick Erickson<erickerickson@gmail.**com< >> erickerick...@gmail.com>> >> >> wrote: >> >> >> >> Well, the short answer is that nobody else has >> >>> 1> had a similar requirement >> >>> AND >> >>> 2> not found a suitable work around >> >>> AND >> >>> 3> implemented the change and contributed it back. >> >>> >> >>> So, if you'd like to volunteer<G>..... >> >>> >> >>> Seriously. If you think this would be valuable and are >> >>> willing to work on it, hop on over to the dev list and >> >>> discuss it, open a JIRA and make it work. I'd start >> >>> by opening a discussion on the dev list before >> >>> opening a JIRA, just to get a sense of where the >> >>> snags would be to changing the Solr code, but that's >> >>> optional. >> >>> >> >>> That said, writing your own update request handler >> >>> that detected this case isn't very difficult, >> >>> extend UpdateRequestProcessorFactory/**UpdateRequestProcessor >> >>> and use it as a plugin. >> >>> >> >>> Best >> >>> Erick >> >>> >> >>> On Wed, Dec 28, 2011 at 6:46 AM, Alexander Aristov >> >>> <alexander.aris...@gmail.com> wrote: >> >>> >> >>>> the problem with dedupe (SignatureUpdateProcessor ) is that it >> REPLACES >> >>>> >> >>> old >> >>> >> >>>> docs. I have tried it already. >> >>>> >> >>>> Best Regards >> >>>> Alexander Aristov >> >>>> >> >>>> >> >>>> On 28 December 2011 13:04, Lance Norskog<goks...@gmail.com> wrote: >> >>>> >> >>>> The SignatureUpdateProcessor is for exactly this problem: >> >>>>> >> >>>>> >> >>>>> >> >>>>> http://www.lucidimagination.**com/search/link?url=http://** >> >>> wiki.apache.org/solr/**Deduplication< >> http://www.lucidimagination.com/search/link?url=http://wiki.apache.org/solr/Deduplication >> > >> >>> >> >>>> On Tue, Dec 27, 2011 at 10:42 PM, Alexander Aristov >> >>>>> <alexander.aris...@gmail.com> wrote: >> >>>>> >> >>>>>> I get docs from external sources and the only place I keep them is >> >>>>>> >> >>>>> solr >> >>> >> >>>> index. I have no a database or other means to track indexed docs (my >> >>>>>> personal oppinion is that it might be a huge headache). >> >>>>>> >> >>>>>> Some docs might change slightly in there original sources but I >> don't >> >>>>>> >> >>>>> need >> >>>>> >> >>>>>> that changes. In fact I need original data only. >> >>>>>> >> >>>>>> So I have no other ways but to either check if a document is already >> >>>>>> >> >>>>> in >> >>> >> >>>> index before I put it to solrj array (read - query solr) or develop my >> >>>>>> >> >>>>> own >> >>>>> >> >>>>>> update chain processor and implement ID check there and skip such >> >>>>>> >> >>>>> docs. >> >>> >> >>>> Maybe it's wrong place to aguee and probably it's been discussed >> >>>>>> >> >>>>> before >> >>> >> >>>> but >> >>>>> >> >>>>>> I wonder why simple the overwrite parameter doesn't work here. >> >>>>>> >> >>>>>> My oppinion it perfectly suits here. In combination with unique ID >> it >> >>>>>> >> >>>>> can >> >>> >> >>>> cover all possible variants. >> >>>>>> >> >>>>>> cases: >> >>>>>> >> >>>>>> 1. overwrite=true and uniquID exists then newer doc should overwrite >> >>>>>> >> >>>>> the >> >>> >> >>>> old one. >> >>>>>> >> >>>>>> 2. overwrite=false and uniqueID exists then newer doc must be >> skipped >> >>>>>> >> >>>>> since >> >>>>> >> >>>>>> old exists. >> >>>>>> >> >>>>>> 3. uniqueID doesn't exist then newer doc just gets added regardless >> if >> >>>>>> >> >>>>> old >> >>>>> >> >>>>>> exists or not. >> >>>>>> >> >>>>>> >> >>>>>> Best Regards >> >>>>>> Alexander Aristov >> >>>>>> >> >>>>>> >> >>>>>> On 27 December 2011 22:53, Erick Erickson<erickerickson@gmail. >> **com<erickerick...@gmail.com> >> >>>>>> > >> >>>>>> >> >>>>> wrote: >> >>>>> >> >>>>>> Mikhail is right as far as I know, the assumption built into Solr is >> >>>>>>> >> >>>>>> that >> >>>>> >> >>>>>> duplicate IDs (when<uniqueKey> is defined) should trigger the old >> >>>>>>> document to be replaced. >> >>>>>>> >> >>>>>>> what is your system-of-record? By that I mean what does your SolrJ >> >>>>>>> program do to send data to Solr? Is there any way you could just >> >>>>>>> *not* send documents that are already in the Solr index based on, >> >>>>>>> for instance, any timestamp associated with your system-of-record >> >>>>>>> and the last time you did an incremental index? >> >>>>>>> >> >>>>>>> Best >> >>>>>>> Erick >> >>>>>>> >> >>>>>>> On Tue, Dec 27, 2011 at 6:38 AM, Alexander Aristov >> >>>>>>> <alexander.aris...@gmail.com> wrote: >> >>>>>>> >> >>>>>>>> Hi >> >>>>>>>> >> >>>>>>>> I am not using database. All needed data is in solr index that's >> >>>>>>>> >> >>>>>>> why I >> >>> >> >>>> want >> >>>>>>> >> >>>>>>>> to skip excessive checks. >> >>>>>>>> >> >>>>>>>> I will check DIH but not sure if it helps. >> >>>>>>>> >> >>>>>>>> I am fluent with Java and it's not a problem for me to write a >> >>>>>>>> >> >>>>>>> class >> >>> >> >>>> or >> >>>>> >> >>>>>> so >> >>>>>>> >> >>>>>>>> but I want to check first maybe there are any ways (workarounds) >> >>>>>>>> >> >>>>>>> to >> >>> >> >>>> make >> >>>>> >> >>>>>> it working without codding, just by playing around with >> >>>>>>>> >> >>>>>>> configuration >> >>> >> >>>> and >> >>>>> >> >>>>>> params. I don't want to go away from default solr implementation. >> >>>>>>>> >> >>>>>>>> Best Regards >> >>>>>>>> Alexander Aristov >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On 27 December 2011 09:33, Mikhail Khludnev< >> >>>>>>>> >> >>>>>>> mkhlud...@griddynamics.com >> >>>>> >> >>>>>> wrote: >> >>>>>>>> >> >>>>>>>> On Tue, Dec 27, 2011 at 12:26 AM, Alexander Aristov< >> >>>>>>>>> alexander.aris...@gmail.com> wrote: >> >>>>>>>>> >> >>>>>>>>> Hi people, >> >>>>>>>>>> >> >>>>>>>>>> I urgently need your help! >> >>>>>>>>>> >> >>>>>>>>>> I have solr 3.3 configured and running. I do uncremental >> >>>>>>>>>> >> >>>>>>>>> indexing 4 >> >>> >> >>>> times a >> >>>>>>>>> >> >>>>>>>>>> day using bulk updates. Some documents are identical to some >> >>>>>>>>>> >> >>>>>>>>> extent >> >>> >> >>>> and I >> >>>>>>> >> >>>>>>>> wish to skip them, not to index. >> >>>>>>>>>> But here is the problem as I could not find a way to tell solr >> >>>>>>>>>> >> >>>>>>>>> ignore >> >>>>> >> >>>>>> new >> >>>>>>> >> >>>>>>>> duplicate docs and keep old indexed docs. I don't care that it's >> >>>>>>>>>> >> >>>>>>>>> new. >> >>>>> >> >>>>>> Just >> >>>>>>>>> >> >>>>>>>>>> determine by ID that such document is in the index already and >> >>>>>>>>>> >> >>>>>>>>> that's >> >>>>> >> >>>>>> it. >> >>>>>>> >> >>>>>>>> I use solrj for indexing. I have tried setting overwrite=false >> >>>>>>>>>> >> >>>>>>>>> and >> >>> >> >>>> dedupe >> >>>>>>> >> >>>>>>>> apprache but nothing helped me. I either have that a newer doc >> >>>>>>>>>> >> >>>>>>>>> overwrites >> >>>>>>> >> >>>>>>>> old one or I get duplicate. >> >>>>>>>>>> >> >>>>>>>>>> I think it's a very simple and basic feature and it must exist. >> >>>>>>>>>> >> >>>>>>>>> What >> >>>>> >> >>>>>> did >> >>>>>>> >> >>>>>>>> I >> >>>>>>>>> >> >>>>>>>>>> make wrong or didn't do? >> >>>>>>>>>> >> >>>>>>>>>> I guess, because the mainstream approach is delta-import , >> when >> >>>>>>>>> >> >>>>>>>> you >> >>> >> >>>> have >> >>>>>>> >> >>>>>>>> "updated" timestamps in your DB and "last-import" timestamp stored >> >>>>>>>>> somewhere. You can check how it works in DIH. >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> Tried google but I couldn't find a solution there althoght many >> >>>>>>>>>> >> >>>>>>>>> people >> >>>>> >> >>>>>> encounted such problem. >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> it's definitely can be done by overriding >> >>>>>>>>> o.a.s.update.**DirectUpdateHandler2.addDoc(**AddUpdateCommand), >> >>>>>>>>> but I >> >>>>>>>>> >> >>>>>>>> suggest >> >>>>>>> >> >>>>>>>> to start from implementing your own >> >>>>>>>>> http://wiki.apache.org/solr/**UpdateRequestProcessor< >> http://wiki.apache.org/solr/UpdateRequestProcessor>- search for >> >>>>>>>>> >> >>>>>>>> PK, >> >>> >> >>>> bypass >> >>>>>>> >> >>>>>>>> chain call if it's found. Then if you meet performance issues on >> >>>>>>>>> >> >>>>>>>> querying >> >>>>>>> >> >>>>>>>> your PKs one by one, (but only after that) you can batch your >> >>>>>>>>> >> >>>>>>>> searches, >> >>>>> >> >>>>>> there are couple of optimization techniques for huge disjunction >> >>>>>>>>> >> >>>>>>>> queries >> >>>>> >> >>>>>> like PK:(2 OR 4 OR 5 OR 6). >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> I start considering that I must query index to check if a doc >> >>>>>>>>>> >> >>>>>>>>> to be >> >>> >> >>>> added >> >>>>>>> >> >>>>>>>> is in the index already and do not add it to array but I have so >> >>>>>>>>>> >> >>>>>>>>> many >> >>>>> >> >>>>>> docs >> >>>>>>>>> >> >>>>>>>>>> that I am affraid it's not a good solution. >> >>>>>>>>>> >> >>>>>>>>>> Best Regards >> >>>>>>>>>> Alexander Aristov >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> -- >> >>>>>>>>> Sincerely yours >> >>>>>>>>> Mikhail Khludnev >> >>>>>>>>> Lucid Certified >> >>>>>>>>> Apache Lucene/Solr Developer >> >>>>>>>>> Grid Dynamics >> >>>>>>>>> >> >>>>>>>>> >> >>>>> >> >>>>> -- >> >>>>> Lance Norskog >> >>>>> goks...@gmail.com >> >>>>> >> >>>>> >> > >> > > > > -- > Sincerely yours > Mikhail Khludnev > Lucid Certified > Apache Lucene/Solr Developer > Grid Dynamics > > <http://www.griddynamics.com> > <mkhlud...@griddynamics.com>
