Re: Solr DataImportHandler

[Shawn Heisey]

On 4/8/2010 2:11 AM, Mark N wrote:
Is it possible to use solr DataImportHandler when that database fields are
not fixed ?  As per my findings we need to configure which table ( entity)
we will read the data and must match which fields in database will map to
fields in solr schema
Since in my case database fields could be dynamic , can DIH be helpful ?

   

You can pass variables into the DIH url that are then used in your query.

Here's what I'm using as the query in my latest config:

SELECT * FROM ${dataimporter.request.dataTable} WHERE did > 
${dataimporter.request.minDid} AND did <= 
${dataimporter.request.maxDid} AND (did % 
${dataimporter.request.numShards}) IN (${dataimporter.request.modVal})"
      deltaQuery="SELECT MAX(did) FROM ${dataimporter.request.dataTable}"
      deltaImportQuery="SELECT * FROM ${dataimporter.request.dataTable} 
WHERE did > ${dataimporter.request.minDid} AND did <= 
${dataimporter.request.maxDid} AND (did % 
${dataimporter.request.numShards}) IN (${dataimporter.request.modVal})

Then here is my URL template:

http://HOST:PORT/solr/CORE/dataimport?command=COMMAND&dataTable=DATATABLE&numShards=NUMSHARDS&modVal=MODVAL&minDid=MINDID&maxDid=MAXDID

And the perl data structure that holds the replacements for the 
uppercase parts:

$urlBits = {
  HOST => $cfg{'shards/inc.host1'},
  PORT => $cfg{'shards/inc.port'},
  MODVAL => $cfg{'shards/inc.modVal'},
  CORE => "live",
  COMMAND => "delta-import&commit=true&optimize=false",
  DATATABLE => $cfg{dataTable},
  NUMSHARDS => $cfg{numShards},
  MINDID => $cfg{maxDid},
  MAXDID => $dbMaxDid,
};