Hey there,
I am doing some hacks to some parts of the solr source. I am doing a feature
for everytime I use delta import hanlder I want it to start geting info from
the db starting from the last indexed document id (from the latest
execution).
The point of doing that is that if I start a full import and the process is
aborted for any reason, I want to be able to start a delta import and start
indexing from the last indexed id of the full import.
To do that basically I have created functions in solrwriter.java and
dataimporter.java. The funcions I have created are the same as the ones to
write and retrieve the timestamp to the dataimport.properties but mines do
it with an id (long instead of date).
I call this functions in docbuilder.java (in the places were functions for
timestamp were created)
I do one more thing... i write in the dataimport.properties every time I
call the function upload in docbuilder to upload a document.
The problem is that not every time the upload function (in docbuilder) is
called a commit is called aswell. So, if I kill -9 the process in the middle
of the execution i will have in the dataimport.properties the last uploaded
id but in the index (opening it with luke) I will have the last commited.
I have done some tests calling writer.commit(false) just after the upload or
setting in solrconfig.xml <maxBufferedDocs>2</maxBufferedDocs>. With both
it works fine but opiously the indexer works extremely slow.
Is there any way to write in the dataimport.properties
(writer.persistIndexLastID(arow.get("id").toString())) just after every
commit but not calling myself the commit function? If not, I would apreciate
any advice about other ways to reach this goals.
If I get it done I will open an issue and upload there the patch cause I
thing that this can be a common use case.
Thanks in advanced
--
View this message in context:
http://www.nabble.com/Delta-import-hack-to-use-last-indexed-id-document-tp20872450p20872450.html
Sent from the Solr - User mailing list archive at Nabble.com.
