Overwriting of column data from DataImportHandler
Hi, I am using solr 1.3 with DataImportHandler from a postgres db. I have a select statement similar to the below Select id, id as pk, name, description from my_table; and a data-config.xml Anyway my issue is that the data thats getting imported into the documents seems to overwriting itself. Basically the pk and the site_id fields is getting api__tweeter__[[id]] where the [[id]] is the id thats returned from the query. Is there something that I am missing? Regards Mark
${dataimporter.delta.twitter_id} not getting populated in deltaImportQuery
Hi,
I have a interesting issue...
Basically I am trying to delta imports on solr 1.4 on a postgresql 8.3
database.
Basically when I am running a delta import with the entity below I get an
exception (see below the entity definition) showing the query its trying to
run and you can see that its not populating the where clause of my
dataImportQuery.
I have tried ${dataimporter.delta.twitter_id} and ${dataimporter.delta.id}
and get the same exceptions.
Am I missing something obvious?
Any help would be appreciated!
Regards
Mark
INFO: Completed parentDeltaQuery for Entity: Tweeter
Nov 11, 2009 3:35:44 PM org.apache.solr.handler.dataimport.DocBuilder
buildDocument
SEVERE: Exception while processing: Tweeter document : SolrInputDocument[{}]
org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
execute query: select twitter_id,twitter_id
as pk,1 as site_id, screen_name from api_tweeter where
twitter_id=;Processing Document # 1
at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:253)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
at
org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:276)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:172)
at
org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:352)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at end of
input
Position: 1197
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:345)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:246)
... 11 more
Nov 11, 2009 3:35:44 PM org.apache.solr.handler.dataimport.DataImporter
doDeltaImport
SEVERE: Delta Import Failed
org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
execute query: select twitter_id,twitter_id
as pk,1 as site_id, screen_name from api_tweeter where
twitter_id=;Processing Document # 1
at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:253)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
at
org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:276)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:172)
at
org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:352)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at end of
input
Position: 1197
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlag
Re: ${dataimporter.delta.twitter_id} not getting populated in deltaImportQuery
I have 2 entities from the root node, not sure if that makes a difference!
On Wed, Nov 11, 2009 at 4:49 PM, Mark Ellul wrote:
> Hi,
>
> I have a interesting issue...
>
> Basically I am trying to delta imports on solr 1.4 on a postgresql 8.3
> database.
>
> Basically when I am running a delta import with the entity below I get an
> exception (see below the entity definition) showing the query its trying to
> run and you can see that its not populating the where clause of my
> dataImportQuery.
>
> I have tried ${dataimporter.delta.twitter_id} and ${dataimporter.delta.id}
> and get the same exceptions.
>
> Am I missing something obvious?
>
> Any help would be appreciated!
>
> Regards
>
> Mark
>
>
> query="
> select twitter_id,
> twitter_id as pk,
> 1 as site_id,
> screen_name
>
> from api_tweeter WHERE
> tweet_mapreduce_on IS NOT NULL;
> " transformer="TemplateTransformer"
>
> deltaImportQuery="
> select twitter_id,
> twitter_id as pk,
> 1 as site_id,
> screen_name
>
> from api_tweeter
> where twitter_id=${dataimporter.delta.twitter_id };
> "
> deltaQuery ="select twitter_id from api_tweeter where modified_on >
> '${dataimporter.last_index_time}' and tweet_mapreduce_on IS NOT NULL;"
>
> >
>
>
>
>
>
>
> INFO: Completed parentDeltaQuery for Entity: Tweeter
> Nov 11, 2009 3:35:44 PM org.apache.solr.handler.dataimport.DocBuilder
> buildDocument
> SEVERE: Exception while processing: Tweeter document :
> SolrInputDocument[{}]
> org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
> execute query: select twitter_id,twitter_id
> as pk,1 as site_id, screen_name from api_tweeter where
> twitter_id=;Processing Document # 1
> at
> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:253)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
> at
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:276)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:172)
> at
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:352)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370)
> Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at end of
> input
> Position: 1197
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
> at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:345)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:246)
> ... 11 more
> Nov 11, 2009 3:35:44 PM org.apache.solr.handler.dataimport.DataImporter
> doDeltaImport
> SEVERE: Delta Import Failed
> org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
> execute query: select twitter_id,twitter_id
> as pk,1 as site_id, screen_name from api_tweeter where
> twitter_id=;Processing Document # 1
> at
> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:253)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
> at
> org.apache.solr.hand
Re: ${dataimporter.delta.twitter_id} not getting populated in deltaImportQuery
Hi Noble,
Thanks for the response.
CAPS is not the issue.
Can you please confirm the link below is the code for the SQLEntityProcessor
in the release 1.4?
http://svn.apache.org/viewvc/lucene/solr/tags/release-1.4.0/contrib/dataimporthandler/src/main/java/org/apache/solr/handler/dataimport/SqlEntityProcessor.java?revision=834197&view=markup
Is there a way to output what is returned from the deltaQuery?
Or the actual queries sent to the database server?
Regards
Mark
2009/11/12 Noble Paul നോബിള് नोब्ळ्
> are you sure the data comes back in the same name. Some DBs return the
> field names in ALL CAPS
>
> you may try out a delta_import using a full import too
>
>
> http://wiki.apache.org/solr/DataImportHandlerFaq#My_delta-import_goes_out_of_memory_._Any_workaround_.3F
>
> On Wed, Nov 11, 2009 at 9:55 PM, Mark Ellul wrote:
> > I have 2 entities from the root node, not sure if that makes a
> difference!
> >
> > On Wed, Nov 11, 2009 at 4:49 PM, Mark Ellul wrote:
> >
> >> Hi,
> >>
> >> I have a interesting issue...
> >>
> >> Basically I am trying to delta imports on solr 1.4 on a postgresql 8.3
> >> database.
> >>
> >> Basically when I am running a delta import with the entity below I get
> an
> >> exception (see below the entity definition) showing the query its
> trying to
> >> run and you can see that its not populating the where clause of my
> >> dataImportQuery.
> >>
> >> I have tried ${dataimporter.delta.twitter_id} and ${
> dataimporter.delta.id}
> >> and get the same exceptions.
> >>
> >> Am I missing something obvious?
> >>
> >> Any help would be appreciated!
> >>
> >> Regards
> >>
> >> Mark
> >>
> >>
> >> >> query="
> >> select twitter_id,
> >> twitter_id as pk,
> >> 1 as site_id,
> >> screen_name
> >>
> >> from api_tweeter WHERE
> >> tweet_mapreduce_on IS NOT NULL;
> >> " transformer="TemplateTransformer"
> >>
> >> deltaImportQuery="
> >> select twitter_id,
> >> twitter_id as pk,
> >> 1 as site_id,
> >> screen_name
> >>
> >> from api_tweeter
> >> where twitter_id=${dataimporter.delta.twitter_id };
> >> "
> >> deltaQuery ="select twitter_id from api_tweeter where modified_on >
> >> '${dataimporter.last_index_time}' and tweet_mapreduce_on IS NOT NULL;"
> >>
> >> >
> >>
> >>
> >>
> >>
> >>
> >>
> >> INFO: Completed parentDeltaQuery for Entity: Tweeter
> >> Nov 11, 2009 3:35:44 PM org.apache.solr.handler.dataimport.DocBuilder
> >> buildDocument
> >> SEVERE: Exception while processing: Tweeter document :
> >> SolrInputDocument[{}]
> >> org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to
> >> execute query: select twitter_id,
> twitter_id
> >> as pk,1 as site_id, screen_name from api_tweeter
> where
> >> twitter_id=;Processing Document # 1
> >> at
> >>
> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
> >> at
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.(JdbcDataSource.java:253)
> >> at
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210)
> >> at
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39)
> >> at
> >>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
> >> at
> >>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:71)
> >> at
> >>
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:237)
> >> at
> >>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:357)
> >> at
> >>
> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:276)
> >> at
> >>
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:172)
> >> at
> >>
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:352)
> >> at
> >>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:391)
> >> at
> >&
$DeleteDocbyQuery in solr 1.4 is not working
Hi,
I have added a deleted field in my database, and am using the
Dataimporthandler to add rows to the index...
I am using solr 1.4
I have added my the deleted field to the query and the RegexTransformer...
and the field definition below
When I run the deltaImport command... I see the below output
INFO: [] webapp=/solr path=/dataimport
params={command=delta-import&debug=true&expungeDeletes=true} status=0
QTime=1
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DataImporter
doDeltaImport
INFO: Starting Delta Import
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
readIndexerProperties
INFO: Read dataimport.properties
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
doDelta
INFO: Starting delta collection.
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Running ModifiedRowKey() for Entity: List
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Creating a connection for entity List with URL:
jdbc:postgresql://localhost:5432/tlists
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Time taken for getConnection(): 4
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Completed ModifiedRowKey for Entity: List rows obtained : 1
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Completed DeletedRowKey for Entity: List rows obtained : 0
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Completed parentDeltaQuery for Entity: List
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
deleteByQuery
INFO: Deleting documents from Solr with query: id:api__list__365522
Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=/mnt/solr-index/index,segFN=segments_r,version=1257863009839,generation=27,filenames=[_bg.fdt,
_bg.tii, segments_r, _bg.fnm, _bg.nrm, _bg.fdx, _bg.prx, _bg.tis, _bg.frq]
Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1257863009839
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
doDelta
INFO: Delta Import completed successfully
It says its deleting the document... but when I do the search its still
showing up
Any Ideas?
Regards
Mark
Re: $DeleteDocbyQuery in solr 1.4 is not working
The doc already existed before the delta-import has been run.
And it exists afterwards... even though it says its deleting it.
Any ideas of what I can try?
On 11/17/09, Noble Paul നോബിള് नोब्ळ् wrote:
> are you sure that the doc w/ the same id was not created after that?
>
> On Mon, Nov 16, 2009 at 11:12 PM, Mark Ellul wrote:
>> Hi,
>>
>> I have added a deleted field in my database, and am using the
>> Dataimporthandler to add rows to the index...
>>
>> I am using solr 1.4
>>
>> I have added my the deleted field to the query and the RegexTransformer...
>> and the field definition below
>>
>> > regex="^true$"
>> replaceWith="id:${List.id}" sourceColName="deleted"/>
>>
>> When I run the deltaImport command... I see the below output
>>
>> INFO: [] webapp=/solr path=/dataimport
>> params={command=delta-import&debug=true&expungeDeletes=true} status=0
>> QTime=1
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DataImporter
>> doDeltaImport
>> INFO: Starting Delta Import
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
>> readIndexerProperties
>> INFO: Read dataimport.properties
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>> doDelta
>> INFO: Starting delta collection.
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>> collectDelta
>> INFO: Running ModifiedRowKey() for Entity: List
>> Nov 16, 2009 5:29:10 PM
>> org.apache.solr.handler.dataimport.JdbcDataSource$1
>> call
>> INFO: Creating a connection for entity List with URL:
>> jdbc:postgresql://localhost:5432/tlists
>> Nov 16, 2009 5:29:10 PM
>> org.apache.solr.handler.dataimport.JdbcDataSource$1
>> call
>> INFO: Time taken for getConnection(): 4
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>> collectDelta
>> INFO: Completed ModifiedRowKey for Entity: List rows obtained : 1
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>> collectDelta
>> INFO: Completed DeletedRowKey for Entity: List rows obtained : 0
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>> collectDelta
>> INFO: Completed parentDeltaQuery for Entity: List
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
>> deleteByQuery
>> INFO: Deleting documents from Solr with query: id:api__list__365522
>> Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy onInit
>> INFO: SolrDeletionPolicy.onInit: commits:num=1
>> commit{dir=/mnt/solr-index/index,segFN=segments_r,version=1257863009839,generation=27,filenames=[_bg.fdt,
>> _bg.tii, segments_r, _bg.fnm, _bg.nrm, _bg.fdx, _bg.prx, _bg.tis, _bg.frq]
>> Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
>> updateCommits
>> INFO: newest commit = 1257863009839
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>> doDelta
>> INFO: Delta Import completed successfully
>>
>> It says its deleting the document... but when I do the search its still
>> showing up
>>
>> Any Ideas?
>>
>> Regards
>>
>> Mark
>>
>
>
>
> --
> -
> Noble Paul | Principal Engineer| AOL | http://aol.com
>
Re: $DeleteDocbyQuery in solr 1.4 is not working
Hi Noble,
I have updated my entity specs, by having a separate entity for
selecting rows which are not deleted for and ones that are deleted, so
I am sure now that the document is not getting added in the same
import.
I read in the tutorial that the deletes are not taken out until the
commit is done. Is there a way I can force a commit? I don't have the
post.jar on the machine in question.
So whats next, I can now safely say that the documents are not getting
added again, and they should be getting deleted... but they not,
unless its not committing and I have to do it manually.
Any other ideas?
Thanks for your help so far!
Regards
Mark
On 11/17/09, Noble Paul നോബിള് नोब्ळ् wrote:
> why don't you add a new timestamp field . you can use the
> TemplateTransformer with the formatDate() function
>
> On Tue, Nov 17, 2009 at 5:49 PM, Mark Ellul wrote:
>> Hi Noble,
>>
>> Excellent Question... should the field that does the deleting be in a
>> different entity to the one that does the addition and updating?
>>
>> If so that could be the issue, I have the field that does the
>> DeleteByQuery command inside of the entity that does the adding.
>>
>> Is there some kind of document metadata where the create date and
>> update date is show?
>>
>> How would I see this meta data if it exists?
>>
>> Regards
>>
>> Mark
>>
>> On 11/17/09, Noble Paul നോബിള് नोब्ळ् wrote:
>>> The question is, did your new delta-import created the doc again?
>>>
>>> On Tue, Nov 17, 2009 at 4:41 PM, Mark Ellul wrote:
>>>> The doc already existed before the delta-import has been run.
>>>>
>>>> And it exists afterwards... even though it says its deleting it.
>>>>
>>>> Any ideas of what I can try?
>>>>
>>>> On 11/17/09, Noble Paul നോബിള് नोब्ळ् wrote:
>>>>> are you sure that the doc w/ the same id was not created after that?
>>>>>
>>>>> On Mon, Nov 16, 2009 at 11:12 PM, Mark Ellul
>>>>> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I have added a deleted field in my database, and am using the
>>>>>> Dataimporthandler to add rows to the index...
>>>>>>
>>>>>> I am using solr 1.4
>>>>>>
>>>>>> I have added my the deleted field to the query and the
>>>>>> RegexTransformer...
>>>>>> and the field definition below
>>>>>>
>>>>>> >>>>> regex="^true$"
>>>>>> replaceWith="id:${List.id}" sourceColName="deleted"/>
>>>>>>
>>>>>> When I run the deltaImport command... I see the below output
>>>>>>
>>>>>> INFO: [] webapp=/solr path=/dataimport
>>>>>> params={command=delta-import&debug=true&expungeDeletes=true} status=0
>>>>>> QTime=1
>>>>>> Nov 16, 2009 5:29:10 PM
>>>>>> org.apache.solr.handler.dataimport.DataImporter
>>>>>> doDeltaImport
>>>>>> INFO: Starting Delta Import
>>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
>>>>>> readIndexerProperties
>>>>>> INFO: Read dataimport.properties
>>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>>>>>> doDelta
>>>>>> INFO: Starting delta collection.
>>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>>>>>> collectDelta
>>>>>> INFO: Running ModifiedRowKey() for Entity: List
>>>>>> Nov 16, 2009 5:29:10 PM
>>>>>> org.apache.solr.handler.dataimport.JdbcDataSource$1
>>>>>> call
>>>>>> INFO: Creating a connection for entity List with URL:
>>>>>> jdbc:postgresql://localhost:5432/tlists
>>>>>> Nov 16, 2009 5:29:10 PM
>>>>>> org.apache.solr.handler.dataimport.JdbcDataSource$1
>>>>>> call
>>>>>> INFO: Time taken for getConnection(): 4
>>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>>>>>> collectDelta
>>>>>> INFO: Completed ModifiedRowKey for Entity: List rows obtained : 1
>>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>>>>>> collectDelta
>&
Re: $DeleteDocbyQuery in solr 1.4 is not working
Thanks Otis... I remember that one!
It still did not remove the document! So obviously its something else thats
happening.
On Tue, Nov 17, 2009 at 10:47 AM, Otis Gospodnetic <
otis_gospodne...@yahoo.com> wrote:
> Mark,
>
> http://localhost:8983/solr/update?stream.body=%3Ccommit/%3E
>
> Otis
> --
> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>
>
>
> - Original Message
> > From: Mark Ellul
> > To: solr-user@lucene.apache.org; noble.p...@gmail.com
> > Sent: Tue, November 17, 2009 9:22:13 AM
> > Subject: Re: $DeleteDocbyQuery in solr 1.4 is not working
> >
> > Hi Noble,
> >
> > I have updated my entity specs, by having a separate entity for
> > selecting rows which are not deleted for and ones that are deleted, so
> > I am sure now that the document is not getting added in the same
> > import.
> >
> > I read in the tutorial that the deletes are not taken out until the
> > commit is done. Is there a way I can force a commit? I don't have the
> > post.jar on the machine in question.
> >
> > So whats next, I can now safely say that the documents are not getting
> > added again, and they should be getting deleted... but they not,
> > unless its not committing and I have to do it manually.
> >
> > Any other ideas?
> >
> > Thanks for your help so far!
> >
> > Regards
> >
> > Mark
> >
> > On 11/17/09, Noble Paul നോബിള് नोब्ळ् wrote:
> > > why don't you add a new timestamp field . you can use the
> > > TemplateTransformer with the formatDate() function
> > >
> > > On Tue, Nov 17, 2009 at 5:49 PM, Mark Ellul wrote:
> > >> Hi Noble,
> > >>
> > >> Excellent Question... should the field that does the deleting be in a
> > >> different entity to the one that does the addition and updating?
> > >>
> > >> If so that could be the issue, I have the field that does the
> > >> DeleteByQuery command inside of the entity that does the adding.
> > >>
> > >> Is there some kind of document metadata where the create date and
> > >> update date is show?
> > >>
> > >> How would I see this meta data if it exists?
> > >>
> > >> Regards
> > >>
> > >> Mark
> > >>
> > >> On 11/17/09, Noble Paul നോബിള് नोब्ळ् wrote:
> > >>> The question is, did your new delta-import created the doc again?
> > >>>
> > >>> On Tue, Nov 17, 2009 at 4:41 PM, Mark Ellul wrote:
> > >>>> The doc already existed before the delta-import has been run.
> > >>>>
> > >>>> And it exists afterwards... even though it says its deleting it.
> > >>>>
> > >>>> Any ideas of what I can try?
> > >>>>
> > >>>> On 11/17/09, Noble Paul നോബിള് नोब्ळ् wrote:
> > >>>>> are you sure that the doc w/ the same id was not created after
> that?
> > >>>>>
> > >>>>> On Mon, Nov 16, 2009 at 11:12 PM, Mark Ellul
> > >>>>> wrote:
> > >>>>>> Hi,
> > >>>>>>
> > >>>>>> I have added a deleted field in my database, and am using the
> > >>>>>> Dataimporthandler to add rows to the index...
> > >>>>>>
> > >>>>>> I am using solr 1.4
> > >>>>>>
> > >>>>>> I have added my the deleted field to the query and the
> > >>>>>> RegexTransformer...
> > >>>>>> and the field definition below
> > >>>>>>
> > >>>>>>
> > >>>>>> regex="^true$"
> > >>>>>> replaceWith="id:${List.id}" sourceColName="deleted"/>
> > >>>>>>
> > >>>>>> When I run the deltaImport command... I see the below output
> > >>>>>>
> > >>>>>> INFO: [] webapp=/solr path=/dataimport
> > >>>>>> params={command=delta-import&debug=true&expungeDeletes=true}
> status=0
> > >>>>>> QTime=1
> > >>>>>> Nov 16, 2009 5:29:10 PM
> > >>>>>> org.apache.solr.handler.dataimport.DataImporter
> > >>>>>> doDeltaImport
> > >>>>>> INFO: Starti
Re: $DeleteDocbyQuery in solr 1.4 is not working
Thanks I will look into it! On Fri, Nov 27, 2009 at 11:34 PM, cpmoser wrote: > > Hi I just recently (well today actually) ran into the same issue, and a > Google search led me here. Something in the log that clued me in to my > issue was this: > > Nov 27, 2009 1:27:05 PM org.apache.solr.core.SolrDeletionPolicy onInit > > I was expecting docs to be removed on commit (which normally happens after > a > dataimport), so I thought something was broken. > > However, when I restarted Solr, the docs that should have been deleted were > actually deleted. There is more discussion about the SolrDeletionPolicy > http://issues.apache.org/jira/browse/SOLR-617 here . I haven't read > enough > to know how to add an onCommit deletion policy to the Solr config yet, and > don't know if you're running into the same issue, but hope this helps. > > > Mark.El wrote: > > > > Thanks Otis... I remember that one! > > > > It still did not remove the document! So obviously its something else > > thats > > happening. > > > > > > > > -- > View this message in context: > http://old.nabble.com/%24DeleteDocbyQuery-in-solr-1.4-is-not-working-tp26376265p26545394.html > Sent from the Solr - User mailing list archive at Nabble.com. > >
