pk vs. uniqueKey with DIH delta-import

[Erik Hatcher]

First - DIH has worked pretty well in a new customer engagement of  
ours.  We've easily imported tens of millions of records with no  
problem.  Kudos to the developers/contributors to DIH - it got us up  
and running quickly.  But now we're delving into more complexities and  
having some issues.

Now on to my current issue, doing a delta-import such that records  
marked as "deleted" in the database are removed from Solr using  
deletedPkQuery.

Here's a config I'm using against a mocked test database:

<dataConfig>
  <dataSource driver="com.mysql.jdbc.Driver" url="jdbc:mysql:// 
localhost/db"/>
  <document name="tests">
    <entity name="test"
            pk="board_id"
            transformer="TemplateTransformer"
            deletedPkQuery="select board_id from boards where deleted  
= 'Y'"
            query="select * from boards where deleted = 'N'"
            deltaImportQuery="select * from boards where deleted = 'N'"
            deltaQuery="select * from boards where deleted = 'N'"
            preImportDeleteQuery="datasource:board">
      <field column="id" template="board-${test.board_id}"/>
      <field column="datasource" template="board"/>
      <field column="title" />
    </entity>
  </document>
</dataConfig>

Note that the uniqueKey in Solr is the "id" field.  And its value is a  
template board-<PK>.

I noticed the javadoc comments in DocBuilder#collectDelta it says  
"Note: In our definition, unique key of Solr document is the primary  
key of the top level entity".  This of course isn't really an  
appropriate assumption.

I also tried a deletedPkQuery of "select concat('board-',board_id)  
from boards where deleted = 'Y'", but got an NPE (relevant stack trace  
below).

It seems that deletedPkQuery only works if the pk and Solr's uniqueKey  
field use the same value.  Is that the case?  If this is the case  
we'll need to fix this somehow.  Any suggestions?

Thanks,
        Erik

stack trace from scenario mentioned above:
SEVERE: Delta Import Failed
java.lang.NullPointerException
	at  
org 
.apache.solr.handler.dataimport.SolrWriter.deleteDoc(SolrWriter.java:83)
	at  
org 
.apache.solr.handler.dataimport.DocBuilder.deleteAll(DocBuilder.java: 
275)
	at  
org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java: 
247)
	at  
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java: 
159)
	at  
org 
.apache 
.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java: 
337)