---------- Forwarded message ----------
From: Noble Paul നോബിള് नोब्ळ् <noble.p...@corp.aol.com>
Date: 2009/11/12
Subject: Re: ${dataimporter.delta.twitter_id} not getting populated in
deltaImportQuery
To: Mark Ellul <m...@catalystic.com>
On Thu, Nov 12, 2009 at 8:17 PM, Mark Ellul <m...@catalystic.com> wrote:
> I think I got it working, thanks for your response... Once I removed the
> TemplateTransformer from the entity.
> Could that have been the issue? Could the template transformer have been
> changing the ${dataimporter.delta.twitter_id} into nothing?
But the though templateTransformer is mentioned , it is not applied on
any field . is it? I do not see the attribute 'template' on any field.
> Regards
> Mark
>
> 2009/11/12 Noble Paul നോബിള് नोब्ळ् <noble.p...@corp.aol.com>
>>
>> delta-import is slightly tricky ,. There are no traps which will let
>> you know the intermediate data. That is why I suggested you to do a
>> delta-import using the full-import .It can probably reveal what is the
>> problem
>>
>> On Thu, Nov 12, 2009 at 6:05 PM, Mark Ellul <m...@catalystic.com> wrote:
>> > Hi Noble,
>> > Thanks for the response.
>> > CAPS is not the issue.
>> > Can you please confirm the link below is the code for the
>> > SQLEntityProcessor
>> > in the release 1.4?
>> >
>> > http://svn.apache.org/viewvc/lucene/solr/tags/release-1.4.0/contrib/dataimporthandler/src/main/java/org/apache/solr/handler/dataimport/SqlEntityProcessor.java?revision=834197&view=markup
>> > Is there a way to output what is returned from the deltaQuery?
>> > Or the actual queries sent to the database server?
>> > Regards
>> > Mark
>> > 2009/11/12 Noble Paul നോബിള് नोब्ळ् <noble.p...@corp.aol.com>
>> >>
>> >> are you sure the data comes back in the same name. Some DBs return the
>> >> field names in ALL CAPS
>> >>
>> >> you may try out a delta_import using a full import too
>> >>
>> >>
>> >>
>> >> http://wiki.apache.org/solr/DataImportHandlerFaq#My_delta-import_goes_out_of_memory_._Any_workaround_.3F
>> >>
>> >> On Wed, Nov 11, 2009 at 9:55 PM, Mark Ellul <m...@catalystic.com>
>> >> wrote:
>> >> > I have 2 entities from the root node, not sure if that makes a
>> >> > difference!
>> >> >
>> >> > On Wed, Nov 11, 2009 at 4:49 PM, Mark Ellul <m...@catalystic.com>
>> >> > wrote:
>> >> >
>> >> >> Hi,
>> >> >>
>> >> >> I have a interesting issue...
>> >> >>
>> >> >> Basically I am trying to delta imports on solr 1.4 on a postgresql
>> >> >> 8.3
>> >> >> database.
>> >> >>
>> >> >> Basically when I am running a delta import with the entity below I
>> >> >> get
>> >> >> an
>> >> >> exception (see below the entity definition) showing the query its
>> >> >> trying to
>> >> >> run and you can see that its not populating the where clause of my
>> >> >> dataImportQuery.
>> >> >>
>> >> >> I have tried ${dataimporter.delta.twitter_id} and
>> >> >> ${dataimporter.delta.id}
>> >> >> and get the same exceptions.
>> >> >>
>> >> >> Am I missing something obvious?
>> >> >>
>> >> >> Any help would be appreciated!
>> >> >>
>> >> >> Regards
>> >> >>
>> >> >> Mark
>> >> >>
>> >> >>
>> >> >> <entity name="Tweeter" pk="twitter_id"
>> >> >> query="
>> >> >> select twitter_id,
>> >> >> twitter_id as pk,
>> >> >> 1 as site_id,
>> >> >> screen_name
>> >> >>
>> >> >> from api_tweeter WHERE
>> >> >> tweet_mapreduce_on IS NOT NULL;
>> >> >> " transformer="TemplateTransformer"
>> >> >>
>> >> >> deltaImportQuery="
>> >> >> select twitter_id,
>> >> >> twitter_id as pk,
>> >> >> 1 as site_id,
>> >> >> screen_name
>> >> >>
>> >> >> from api_tweeter
>> >> >> where twitter_id=${dataimporter.delta.twitter_id };
>> >> >> "
>> >> >> deltaQuery ="select twitter_id from api_tweeter where modified_on >
>> >> >> '${dataimporter.last_index_time}' and tweet_mapreduce_on IS NOT
>> >> >> NULL;"
>> >> >>
>> >> >> >
>> >> >> <field name="twitter_id" column="twitter_id" />
>> >> >>
>> >> >>
>> >> >> </entity>
>> >> >>
>> >> >>
>> >> >> INFO: Completed parentDeltaQuery for Entity: Tweeter
>> >> >> Nov 11, 2009 3:35:44 PM
>> >> >> org.apache.solr.handler.dataimport.DocBuilder
>> >> >> buildDocument
>> >> >> SEVERE: Exception while processing: Tweeter document :
>> >> >> SolrInputDocument[{}]
>> >> >> org.apache.solr.handler.dataimport.DataImportHandlerException:
>> >> >> Unable
>> >> >> to
>> >> >> execute query: select twitter_id,
>> >> >> twitter_id
>> >> >> as pk, 1 as site_id, screen_name from api_tweeter
>> >> >> where
>> >> >> twitter_id=; Processing Document # 1
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:253)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:276)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:172)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:352)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
>> >> >> Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at
>> >> >> end of
>> >> >> input
>> >> >> Position: 1197
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:345)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:246)
>> >> >> ... 11 more
>> >> >> Nov 11, 2009 3:35:44 PM
>> >> >> org.apache.solr.handler.dataimport.DataImporter
>> >> >> doDeltaImport
>> >> >> SEVERE: Delta Import Failed
>> >> >> org.apache.solr.handler.dataimport.DataImportHandlerException:
>> >> >> Unable
>> >> >> to
>> >> >> execute query: select twitter_id,
>> >> >> twitter_id
>> >> >> as pk, 1 as site_id, screen_name from api_tweeter
>> >> >> where
>> >> >> twitter_id=; Processing Document # 1
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:253)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:276)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:172)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:352)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
>> >> >> Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at
>> >> >> end of
>> >> >> input
>> >> >> Position: 1197
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:345)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:246)
>> >> >> ... 11 more
>> >> >> Nov 11, 2009 3:35:44 PM org.apache.solr.update.DirectUpdateHandler2
>> >> >> rollback
>> >> >> INFO: start rollback
>> >> >> Nov 11, 2009 3:35:44 PM org.apache.solr.update.DirectUpdateHandler2
>> >> >> rollback
>> >> >> INFO: end_rollback
>> >> >>
>> >> >>
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> -----------------------------------------------------
>> >> Noble Paul | Principal Engineer| AOL | http://aol.com
>> >
>> >
>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul | Principal Engineer| AOL | http://aol.com
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
