Why write a Perl script for that? touch new_timestamp find . -newer timestamp | script-to-submit && mv new_timestamp timestamp
Neither approach deals with deleted files. To do this correctly, you need lists of all the files in the index with their timestamps, and of all the files in the repository. Then you need to difference them to find deleted ones, new ones, and ones that have changed. You might even want to track links and symlinks to get dupes and canonical paths. wunder On Feb 12, 2014, at 10:00 PM, "Sadler, Anthony" <anthony.sad...@yrgrp.com> wrote: > I had this problem when I started to look at Solr as an index for a file > server. What I ended up doing was writing a perl script that did this: > > - Scan the whole filesystem and create an XML that is submitted into Solr for > indexing. As this might be some 600,000 files, I break it down into chunks of > N files (N = 200 currently). > - At the end of a successful scan, write out the time it started to a file. > - Next time you run the script, the script looks for the start time file. It > reads that in and checks every file in the system: > = If it has a mod_time greater than the begin_time, it has changed since we > last updated it, so reindex it. > = If it doesn't, just update the last_seen timestamp in Solr (a field we > created) so we know its still there. > > We're doing that and its indexing just fine. > > -----Original Message----- > From: lalitjangra [mailto:lalit.j.jan...@gmail.com] > Sent: Thursday, 13 February 2014 4:45 PM > To: solr-user@lucene.apache.org > Subject: Re: Solr delta indexing approach > > Thanks Alex, > > Yes my source system maintains the crettion & last modificaiton system of > each document. > > As per your inputs, can i assume that next time when solr starts indexing, it > scans all the prsent in source but only picks those for indexing which are > either new or have been updated since last successful indexing. > > How solr does this or in short what is solr strategy for indexing? I would > definitely like to know more about it & if you can share your thoughts on > same, it would be great. > > Regards, > Lalit. > > > > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Solr-delta-indexing-approach-tp4117068p4117077.html > Sent from the Solr - User mailing list archive at Nabble.com. > > > ________________________________ > > ========================================== > Privileged/Confidential Information may be contained in this message. If you > are not the addressee indicated in this message (or responsible for delivery > of the message to such person), you may not copy or deliver this message to > anyone. In such case, you should destroy this message and kindly notify the > sender by reply email. Please advise immediately if you or your employer does > not consent to email for messages of this kind. Opinions, conclusions and > other information in this message that do not relate to the official business > of Burson-Marsteller shall be understood as neither given nor endorsed by it. > ========================================== > -- Walter Underwood wun...@wunderwood.org
