Hi Shalin,
I am using apache-solr-1.3.0 in my project. I thought it to be the latest
release. Does it not support delta import?
Thanks,
Manu
Shalin Shekhar Mangar wrote:
>
> Which version of Solr are you using? The deltaImportQuery feature was
> added
> recently, you'd need a nightly build for it to work.
>
> On Mon, Dec 29, 2008 at 4:57 PM, Manupriya
> <manupriya.si...@gmail.com>wrote:
>
>>
>> Hi Noble,
>>
>> I tried with the deltaImportQuery as well. But still I am getting the
>> same
>> exception on the server console -
>>
>> Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
>> doDeltaImport
>>
>> SEVERE: Delta Import Failed
>> java.lang.NullPointerException
>> at
>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>> at
>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>> at
>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>> at
>>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>> at
>>
>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>> at
>>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>> at
>>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>
>> Another thing, I do have proper data in the DB.
>>
>> Thanks,
>> Manu
>>
>>
>> Noble Paul നോബിള് नोब्ळ् wrote:
>> >
>> > The same example with deltaImportQuery would look as follows
>> > <dataConfig>
>> > <dataSource driver="com.mysql.jdbc.Driver"
>> > url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root"
>> />
>> > <document name="products">
>> > <entity name="item" pk="ID" query="select * from item"
>> > deltaQuery="select id from item where last_modified >
>> > '${dataimporter.last_index_time}'"
>> > deltaImportQuery="select * from item where ID=${dataimporter.delta.ID}"
>> >>
>> > <field column="NAME" name="name" />
>> > <field column="NAME" name="nameSort" />
>> > <field column="NAME" name="alphaNameSort" />
>> >
>> > <entity name="feature" pk="ITEM_ID"
>> > query="select DESCRIPTION from FEATURE where
>> > ITEM_ID='${item.ID}'"
>> > deltaQuery="select ITEM_ID from FEATURE where
>> > last_modified > '${dataimporter.last_index_time}'"
>> > parentDeltaQuery="select ID from item where
>> > ID=${feature.ITEM_ID}">
>> > <field name="features" column="DESCRIPTION" />
>> > </entity>
>> >
>> > <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
>> > query="select CATEGORY_ID from item_category where
>> > ITEM_ID='${item.ID}'"
>> > deltaQuery="select ITEM_ID, CATEGORY_ID from
>> > item_category where last_modified > '${dataimporter.last_index_time}'"
>> > parentDeltaQuery="select ID from item where
>> > ID=${item_category.ITEM_ID}">
>> > <entity name="category" pk="ID"
>> > query="select DESCRIPTION from category where ID
>> =
>> > '${item_category.CATEGORY_ID}'"
>> > deltaQuery="select ID from category where
>> > last_modified > '${dataimporter.last_index_time}'"
>> > parentDeltaQuery="select ITEM_ID, CATEGORY_ID
>> from
>> > item_category where CATEGORY_ID=${category.ID}">
>> > <field column="description" name="cat" />
>> > </entity>
>> > </entity>
>> > </entity>
>> > </document>
>> > </dataConfig>
>> >
>> > On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള് नोब्ळ्
>> > <noble.p...@gmail.com> wrote:
>> >> did you put in the data?
>> >>
>> >> On Mon, Dec 29, 2008 at 3:07 PM, Manupriya <manupriya.si...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi Shalin,
>> >>>
>> >>> I am actually trying out the example provided at
>> >>> http://wiki.apache.org/solr/DataImportHandler.
>> >>>
>> >>> My db-data-config.xml is as following -
>> >>>
>> >>> <dataConfig>
>> >>> <dataSource driver="com.mysql.jdbc.Driver"
>> >>> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root"
>> />
>> >>> <document name="products">
>> >>> <entity name="item" pk="ID" query="select * from item"
>> >>> deltaQuery="select id from item where last_modified >
>> >>> '${dataimporter.last_index_time}'">
>> >>> <field column="NAME" name="name" />
>> >>> <field column="NAME" name="nameSort" />
>> >>> <field column="NAME" name="alphaNameSort" />
>> >>>
>> >>> <entity name="feature" pk="ITEM_ID"
>> >>> query="select DESCRIPTION from FEATURE where
>> >>> ITEM_ID='${item.ID}'"
>> >>> deltaQuery="select ITEM_ID from FEATURE where
>> >>> last_modified > '${dataimporter.last_index_time}'"
>> >>> parentDeltaQuery="select ID from item where
>> >>> ID=${feature.ITEM_ID}">
>> >>> <field name="features" column="DESCRIPTION" />
>> >>> </entity>
>> >>>
>> >>> <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
>> >>> query="select CATEGORY_ID from item_category where
>> >>> ITEM_ID='${item.ID}'"
>> >>> deltaQuery="select ITEM_ID, CATEGORY_ID from
>> >>> item_category where last_modified >
>> '${dataimporter.last_index_time}'"
>> >>> parentDeltaQuery="select ID from item where
>> >>> ID=${item_category.ITEM_ID}">
>> >>> <entity name="category" pk="ID"
>> >>> query="select DESCRIPTION from category where
>> ID
>> >>> =
>> >>> '${item_category.CATEGORY_ID}'"
>> >>> deltaQuery="select ID from category where
>> >>> last_modified > '${dataimporter.last_index_time}'"
>> >>> parentDeltaQuery="select ITEM_ID, CATEGORY_ID
>> >>> from
>> >>> item_category where CATEGORY_ID=${category.ID}">
>> >>> <field column="description" name="cat" />
>> >>> </entity>
>> >>> </entity>
>> >>> </entity>
>> >>> </document>
>> >>> </dataConfig>
>> >>>
>> ---------------------------------------------------------------------------------------------------
>> >>>
>> >>> My DB structure is as following -
>> >>>
>> >>> DROP TABLE IF EXISTS solr_demo_db.item;
>> >>> CREATE TABLE `item` (
>> >>> `ID` int(2) NOT NULL auto_increment,
>> >>> `name` varchar(100) default NULL,
>> >>> `manu` varchar(20) default NULL,
>> >>> `weight` varchar(20) default NULL,
>> >>> `price` varchar(20) default NULL,
>> >>> `popularity` varchar(2) default NULL,
>> >>> `includes` varchar(10) default NULL,
>> >>> `last_modified` datetime default NULL,
>> >>> PRIMARY KEY (`ID`)
>> >>> ) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
>> >>>
>> >>> DROP TABLE IF EXISTS solr_demo_db.feature;
>> >>> CREATE TABLE `feature` (
>> >>> `DESCRIPTION` varchar(100) default NULL,
>> >>> `ITEM_ID` int(2) NOT NULL,
>> >>> `last_modified` datetime default NULL,
>> >>> PRIMARY KEY (`ITEM_ID`)
>> >>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>> >>>
>> >>> DROP TABLE IF EXISTS solr_demo_db.category;
>> >>> CREATE TABLE `category` (
>> >>> `ID` int(2) NOT NULL auto_increment,
>> >>> `DESCRIPTION` varchar(100) default NULL,
>> >>> `last_modified` date default NULL,
>> >>> PRIMARY KEY (`ID`)
>> >>> ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
>> >>>
>> >>> DROP TABLE IF EXISTS solr_demo_db.item_category;
>> >>> CREATE TABLE `item_category` (
>> >>> `ITEM_ID` int(2) NOT NULL default '0',
>> >>> `CATEGORY_ID` int(2) NOT NULL default '0',
>> >>> `last_modified` date default NULL,
>> >>> PRIMARY KEY (`ITEM_ID`,`CATEGORY_ID`)
>> >>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>> >>>
>> >>> Am I doing something wrong here?
>> >>>
>> >>> Thanks,
>> >>> Manu
>> >>>
>> >>>
>> >>> Shalin Shekhar Mangar wrote:
>> >>>>
>> >>>> What does your data-config look like? especially the delta query
>> part.
>> >>>>
>> >>>> On Mon, Dec 29, 2008 at 2:35 PM, Manupriya
>> >>>> <manupriya.si...@gmail.com>wrote:
>> >>>>
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> I am using Delta DataImport feature to partially refresh the
>> indexes.
>> >>>>>
>> >>>>> Note - full-import is working perfectly fine. But I need to do
>> >>>>> delta-import
>> >>>>> as I do not want to rebuild all the indexes.
>> >>>>>
>> >>>>> My database structure is same as mentioned in the example at
>> >>>>> http://wiki.apache.org/solr/DataImportHandler.
>> >>>>>
>> >>>>> http://www.nabble.com/file/p21200498/example-schema.png
>> >>>>>
>> ---------------------------------------------------------------------------
>> >>>>> The dataimport.properties has the value as -
>> >>>>>
>> >>>>> #Mon Dec 29 14:08:12 IST 2008
>> >>>>> last_index_time=2008-12-29 14\:08\:12
>> >>>>>
>> ---------------------------------------------------------------------------
>> >>>>> I have changed the last_modified for the 'tem' table as current
>> >>>>> timestamp.
>> >>>>> And when I query the DB, I get the following result -
>> >>>>>
>> >>>>> http://www.nabble.com/file/p21200498/data.jpg
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------------------------
>> >>>>>
>> >>>>> But when I call delta data-import, it returns me the following
>> >>>>> response -
>> >>>>>
>> >>>>> Server Response:
>> >>>>>
>> >>>>>
>> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
>> >>>>> response format is experimental. It is likely to change in the
>> >>>>> future.}
>> >>>>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------------------------------------
>> >>>>> And when I query for data for '*:*', I get the following -
>> >>>>> Note - Data returned by query doesnt reflect the correct timestamp
>> >>>>> from
>> >>>>> database.
>> >>>>>
>> >>>>> SolrDocument(1)
>> >>>>> Field Name & Value: includes=12
>> >>>>> Field Name & Value: features=[demo feature, demo feature 1]
>> >>>>> Field Name & Value: price=12.0
>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> >>>>> Field Name & Value: manu=manu-12
>> >>>>> Field Name & Value: sku=1
>> >>>>> Field Name & Value: id=1
>> >>>>> Field Name & Value: popularity=1
>> >>>>> Field Name & Value: weight=12.0
>> >>>>> SolrDocument(2)
>> >>>>> Field Name & Value: includes=43
>> >>>>> Field Name & Value: features=demo feature 2
>> >>>>> Field Name & Value: price=10.0
>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> >>>>> Field Name & Value: manu=Demo - 12
>> >>>>> Field Name & Value: sku=2
>> >>>>> Field Name & Value: id=2
>> >>>>> Field Name & Value: popularity=5
>> >>>>> Field Name & Value: weight=12.0
>> >>>>> SolrDocument(3)
>> >>>>> Field Name & Value: includes=1
>> >>>>> Field Name & Value: features=demo -3
>> >>>>> Field Name & Value: price=1.0
>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> >>>>> Field Name & Value: manu=manu - 36
>> >>>>> Field Name & Value: sku=3
>> >>>>> Field Name & Value: id=3
>> >>>>> Field Name & Value: popularity=1
>> >>>>> Field Name & Value: weight=1.0
>> >>>>> SolrDocument(4)
>> >>>>> Field Name & Value: includes=2
>> >>>>> Field Name & Value: features=demo - 4
>> >>>>> Field Name & Value: price=2.0
>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>> >>>>> Field Name & Value: manu=manu - 46
>> >>>>> Field Name & Value: sku=4
>> >>>>> Field Name & Value: id=4
>> >>>>> Field Name & Value: popularity=2
>> >>>>> Field Name & Value: weight=2.0
>> >>>>> SolrDocument(5)
>> >>>>> Field Name & Value: includes=3
>> >>>>> Field Name & Value: features=demo - 5
>> >>>>> Field Name & Value: price=3.0
>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>> >>>>> Field Name & Value: manu=manu - 56
>> >>>>> Field Name & Value: sku=5
>> >>>>> Field Name & Value: id=5
>> >>>>> Field Name & Value: popularity=3
>> >>>>> Field Name & Value: weight=3.0
>> >>>>> SolrDocument(6)
>> >>>>> Field Name & Value: includes=4
>> >>>>> Field Name & Value: features=demo - 66
>> >>>>> Field Name & Value: price=4.0
>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>> >>>>> Field Name & Value: manu=manu - 66
>> >>>>> Field Name & Value: sku=6
>> >>>>> Field Name & Value: id=6
>> >>>>> Field Name & Value: popularity=4
>> >>>>> Field Name & Value: weight=4.0
>> >>>>>
>> >>>>>
>> >>>>>
>> ------------------------------------------------------------------------------------------
>> >>>>> My Java code that calls delta import is as follows -
>> >>>>>
>> >>>>> public static void main(String[] args) throws IOException,
>> >>>>> SolrServerException {
>> >>>>> CommonsHttpSolrServer server = new
>> >>>>> CommonsHttpSolrServer("http://localhost:8983/solr";);
>> >>>>> BinaryResponseParser parser = new BinaryResponseParser();
>> >>>>> server.setParser(parser);
>> >>>>>
>> >>>>> ModifiableSolrParams params = new ModifiableSolrParams();
>> >>>>> params.set("qt", "/dataimport");
>> >>>>> params.set("command", "delta-import");
>> >>>>> params.set("commit", "true");
>> >>>>> params.set("wt","json");
>> >>>>>
>> >>>>> try{
>> >>>>>
>> >>>>> QueryResponse response = server.query(params);
>> >>>>> server.commit();
>> >>>>> System.out.println("Server Response: " + response);
>> >>>>>
>> >>>>> SolrQuery query = new SolrQuery("*:*");
>> >>>>> QueryResponse results = server.query(query);
>> >>>>>
>> >>>>> SolrDocumentList list = results.getResults();
>> >>>>>
>> >>>>> int index = 1;
>> >>>>> for (SolrDocument solrDocument : list) {
>> >>>>>
>> >>>>> System.out.println("SolrDocument("+index+")");
>> >>>>> Iterator<Entry<String, Object>> iterator =
>> >>>>> solrDocument.iterator();
>> >>>>>
>> >>>>> while(iterator.hasNext()){
>> >>>>> System.out.println("Field Name & Value:
>> >>>>> "+iterator.next());
>> >>>>> }
>> >>>>>
>> >>>>> index++;
>> >>>>> }
>> >>>>>
>> >>>>> }catch(Exception ex){
>> >>>>>
>> >>>>> System.out.println("Exception Occured:"+ex);
>> >>>>> }
>> >>>>>
>> >>>>> }
>> >>>>>
>> ---------------------------------------------------------------------------
>> >>>>>
>> >>>>> Does SOLR do some kind of caching? I dont understand as why the
>> >>>>> updated
>> >>>>> values are not queried correctly from DB?
>> >>>>>
>> >>>>> There are no errors in the logs. But I can see an error on the
>> server
>> >>>>> console -
>> >>>>>
>> >>>>> SEVERE: Delta Import Failed
>> >>>>> java.lang.NullPointerException
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>> >>>>> at
>> >>>>>
>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>> >>>>> at
>> >>>>>
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>> >>>>>
>> >>>>>
>> >>>>> What can be the reason for this error?
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Manu
>> >>>>>
>> >>>>> --
>> >>>>> View this message in context:
>> >>>>>
>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200498.html
>> >>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Regards,
>> >>>> Shalin Shekhar Mangar.
>> >>>>
>> >>>>
>> >>>
>> >>> --
>> >>> View this message in context:
>> >>>
>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200786.html
>> >>> Sent from the Solr - User mailing list archive at Nabble.com.
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> --Noble Paul
>> >>
>> >
>> >
>> >
>> > --
>> > --Noble Paul
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21201980.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context:
http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21202107.html
Sent from the Solr - User mailing list archive at Nabble.com.
