> The slave is polling the master after the interval specified in solrconfig.xml. The slave essentially asks "has anything changed?" If so, the changes are brought down to the slave. Yes, I understand this, but if master does not open a new searcher after auto commits (which would indicate that the new index is not quite ready yet) and if master is still using the old index to serve search requests, I would expect the slave to do the same as well. Or the slave should at least not replicate or not open a new searcher, until the master opened a new searcher. But that is just the way I see it and it may be wrong.
> What's your polling interval on the slave anyway? Sounds like it's quite frequent if you notice this immediately after the DIH starts. No, polling interval is set to 1 hour, but the full import was set to run at 1 AM. I believe a delete followed by few docs got replicated after the first few auto commits when the slave probably polled around 1:10 AM and slave index had few docs for an hour before the next polling happened, which is why the date query was returning empty results for exactly that one hour. (The full index takes about 1.5 hours to finish.) Anyway the problem is now solved by specifying "clean=false" in the DIH full import command. On Sat, Mar 1, 2014 at 9:12 AM, Erick Erickson <erickerick...@gmail.com>wrote: > bq: the slave anyway replicates the index after auto commits! (Is this > desired behavior?) > > Absolutely it's desired behavior. The slave is polling the master > after the interval > specified in solrconfig.xml. The slave essentially asks "has anything > changed?" If so, > the changes are brought down to the slave. And by definition, commits > change the index, > especially if all docs have been deleted.... > > What's your polling interval on the slave anyway? Sounds like it's > quite frequent if you > notice this immediately after the DIH starts. > > Best, > Erick > > On Fri, Feb 28, 2014 at 9:04 PM, Arun Rangarajan > <arunrangara...@gmail.com> wrote: > > I believe I figured out what the issue is. Even though we do not open a > new > > searcher on master during full import, the slave anyway replicates the > > index after auto commits! (Is this desired behavior?) Since "clean=true" > > this meant all the docs were deleted on slave and a partial index got > > replicated! The reason only the date query did not return any results is > > because recently created docs have higher doc IDs and we index by > ascending > > order of IDs! > > > > I believe I have two options: > > - as Chris suggested I have to use "clean=false" so the existing docs are > > not deleted first on the slave. Since we have primary keys, newly added > > docs will overwrite old docs as they get added. > > - disable replication after commits. Replicate only after optimize. > > > > Thx all for your help. > > > > > > > > > > > > On Fri, Feb 28, 2014 at 8:06 PM, Arun Rangarajan > > <arunrangara...@gmail.com>wrote: > > > >> Thx, Erick and Chris. > >> > >> This is indeed very strange. Other queries which do not restrict by the > >> date field are returning results, so the index is definitely not empty. > Has > >> it got something to do with the date query part, with NOW/DAY or > something > >> in here? > >> first_publish_date:[NOW/DAY-33DAYS TO NOW/DAY-3DAYS] > >> > >> For now, I have set up a script to just log the number of docs on the > >> slave every minute. Will monitor and report the findings. > >> > >> > >> On Fri, Feb 28, 2014 at 6:49 PM, Chris Hostetter < > hossman_luc...@fucit.org > >> > wrote: > >> > >>> > >>> : This is odd. The full import, I think, deletes the > >>> : docs in the index when it starts. > >>> > >>> Yeah, if you are doing a full-import everyday, and you don't want it to > >>> delete all docs when it starts, you need to specify "clearn=false" > >>> > >>> > >>> > https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler#UploadingStructuredDataStoreDatawiththeDataImportHandler-Parametersforthefull-importCommand > >>> > >>> > >>> > >>> -Hoss > >>> http://www.lucidworks.com/ > >>> > >> > >> >
