dataimport.properties is not updated on delta-import

[warb]

Hello!

I am having some difficulties getting dataimport (DIH) to behave correctly
in Solr 1.4.0. Indexing itself works just as it is supposed to with both
full-import and delta-import adding modified or newly created records to the
index. The problem is however that the date and time of the last
delta-import is not updated in the "dataimport.properites" file. The only
time the file gets updated is when performing a full-import. 

Now, this is not a huge problem since delta-import will simply disregard
records already imported (due to the primary key), but it seems wasteful to
fetch records which have already been added on previous runs. Also, as the
database grows the delta-imports will take longer and longer.

Does anyone know of anything I might have overlooked or known bugs?

Thanks in advance!

Johan Andersson
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/dataimport-properties-is-not-updated-on-delta-import-tp916753p916753.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: dataimport.properties is not updated on delta-import

[warb]

Hello again!

Upon further investigation it seems that something is amiss with
delta-import after all, the delta-import does not actually import anything
(I thought it did when I ran it previously but I am not sure that was the
case any longer.) It does complete successfully as seen from the front-end
(dataimport?command=delta-import). Also in the logs it is stated the the
import was successful (INFO: Delta Import completed successfully), but there
are exception pertaining to some documents. 

The exception message is that the id field is missing
(org.apache.solr.common.SolrException: Document [null] missing required
field: id). Now, I have checked the column names in the table, the
data-config.xml file and the schema.xml file and they all have the
column/field names written in lowercase and are even named exactly the same.

Do Solr rollback delta-imports if one or more of the documents failed?
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/dataimport-properties-is-not-updated-on-delta-import-tp916753p919609.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: dataimport.properties is not updated on delta-import

[warb]

I've finally found the problem causing the delta-import to fail and thought I
would post it here for future reference (if someone makes the same mistake I
did).

I had forgot to collect the "id" column in the deltaImportQuery. I should,
of course, have known this from the log entires about documents not being
added because they lacked "id" fields.

My broken query: SELECT column_1, column_2, column_3 FROM table WHERE id =
${dataimporter.delta.id}
My working query: SELECT id, column_1, column_2, column_3 FROM table WHERE
id = ${dataimporter.delta.id}

Thanks all for you help!
-- 
View this message in context: 
http://lucene.472066.n3.nabble.com/dataimport-properties-is-not-updated-on-delta-import-tp916753p933342.html
Sent from the Solr - User mailing list archive at Nabble.com.