I had this problem when I started to look at Solr as an index for a file server. What I ended up doing was writing a perl script that did this:
- Scan the whole filesystem and create an XML that is submitted into Solr for indexing. As this might be some 600,000 files, I break it down into chunks of N files (N = 200 currently). - At the end of a successful scan, write out the time it started to a file. - Next time you run the script, the script looks for the start time file. It reads that in and checks every file in the system: = If it has a mod_time greater than the begin_time, it has changed since we last updated it, so reindex it. = If it doesn't, just update the last_seen timestamp in Solr (a field we created) so we know its still there. We're doing that and its indexing just fine. -----Original Message----- From: lalitjangra [mailto:lalit.j.jan...@gmail.com] Sent: Thursday, 13 February 2014 4:45 PM To: solr-user@lucene.apache.org Subject: Re: Solr delta indexing approach Thanks Alex, Yes my source system maintains the crettion & last modificaiton system of each document. As per your inputs, can i assume that next time when solr starts indexing, it scans all the prsent in source but only picks those for indexing which are either new or have been updated since last successful indexing. How solr does this or in short what is solr strategy for indexing? I would definitely like to know more about it & if you can share your thoughts on same, it would be great. Regards, Lalit. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-delta-indexing-approach-tp4117068p4117077.html Sent from the Solr - User mailing list archive at Nabble.com. ________________________________ ========================================== Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to email for messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of Burson-Marsteller shall be understood as neither given nor endorsed by it. ==========================================
