At the risk of derailing the thread: We do a lot more in the script than is mentioned here: We pull out parts of the path and mangle them (for example turn them into a UNC path for users to use, or pull out a client name or job number using a known folder structure). As for deleted files, here's how the script works in totality:
- Script runs first time, finds every file and puts into formerly empty Solr DB. For every file found, set date_last_seen = current_time. Writes out last_begin_time file and ends. - Secondary function of script runs sometime after, looks for any file with a date_last_seen < last_begin_time. Nothing is found this time around. - Script runs next time, see's that there is a last_begin_time file, reads in that time. Script then runs in a "delta" mode, looking for all files modified later than last_begin_time. If it finds them, it re-indexs them and their contents. All other files that have mod_time less than last_run_time merely have their date_last_seen updated. Script ends, writes out last_begin_time file. At this point, any files that were deleted between the first and second run were not updated , so their date_last_seen is different from all the others. This gives me something to look for. - Secondary function of script runs sometime after, looks for any file with a date_last_seen < last_begin_time. This time around, some files are found. These files have their isDeleted field in solr set to 1. Hopefully that makes a bit more sense. -----Original Message----- From: Walter Underwood [mailto:wun...@wunderwood.org] Sent: Thursday, 13 February 2014 5:26 PM To: solr-user@lucene.apache.org Subject: Re: Solr delta indexing approach Why write a Perl script for that? touch new_timestamp find . -newer timestamp | script-to-submit && mv new_timestamp timestamp Neither approach deals with deleted files. To do this correctly, you need lists of all the files in the index with their timestamps, and of all the files in the repository. Then you need to difference them to find deleted ones, new ones, and ones that have changed. You might even want to track links and symlinks to get dupes and canonical paths. wunder On Feb 12, 2014, at 10:00 PM, "Sadler, Anthony" <anthony.sad...@yrgrp.com> wrote: > I had this problem when I started to look at Solr as an index for a file > server. What I ended up doing was writing a perl script that did this: > > - Scan the whole filesystem and create an XML that is submitted into Solr for > indexing. As this might be some 600,000 files, I break it down into chunks of > N files (N = 200 currently). > - At the end of a successful scan, write out the time it started to a file. > - Next time you run the script, the script looks for the start time file. It > reads that in and checks every file in the system: > = If it has a mod_time greater than the begin_time, it has changed since we > last updated it, so reindex it. > = If it doesn't, just update the last_seen timestamp in Solr (a field we > created) so we know its still there. > > We're doing that and its indexing just fine. > > -----Original Message----- > From: lalitjangra [mailto:lalit.j.jan...@gmail.com] > Sent: Thursday, 13 February 2014 4:45 PM > To: solr-user@lucene.apache.org > Subject: Re: Solr delta indexing approach > > Thanks Alex, > > Yes my source system maintains the crettion & last modificaiton system of > each document. > > As per your inputs, can i assume that next time when solr starts indexing, it > scans all the prsent in source but only picks those for indexing which are > either new or have been updated since last successful indexing. > > How solr does this or in short what is solr strategy for indexing? I would > definitely like to know more about it & if you can share your thoughts on > same, it would be great. > > Regards, > Lalit. > > > > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Solr-delta-indexing-approach-tp4117 > 068p4117077.html Sent from the Solr - User mailing list archive at > Nabble.com. > > > ________________________________ > > ========================================== > Privileged/Confidential Information may be contained in this message. If you > are not the addressee indicated in this message (or responsible for delivery > of the message to such person), you may not copy or deliver this message to > anyone. In such case, you should destroy this message and kindly notify the > sender by reply email. Please advise immediately if you or your employer does > not consent to email for messages of this kind. Opinions, conclusions and > other information in this message that do not relate to the official business > of Burson-Marsteller shall be understood as neither given nor endorsed by it. > ========================================== > -- Walter Underwood wun...@wunderwood.org
