Select and Update after with DataImportHandler

[Gabi]

Hello, we have a sql to retrieve fields from DB and they are indexed in a 
full-import defined in a data-config.xml.

It works as expected but now we want to update the database fields so we know 
when data was imported.
So we add to the table being indexed a timestamp. Now we have table1 field1 
field2... and fieldN (the timestamp).

Is there any way to indicate "after you finish the select of the full-import, 
now you have to do this sql update of the timestamp in the database table"?

We use Solr 4.6.0


We've tried in the same sql, with cursors or transactions, etc, but the sql 
select is pretty complex and we can't make it work in such way, but perhaps the 
only way to do it is to substitute the select with some kind of transaction but 
we don’t know if this is possible to do in the query for the full-import.

Regards.


---
El software de antivirus Avast ha analizado este correo electrónico en busca de 
virus.
http://www.avast.com

RE: Select and Update after with DataImportHandler

[Gabi]

Yes, that's the point Steve.

In a full-import from the UI of Solr a select defined in configuration file
data-config.xml is executed.
This select retrieves the fields indexed as defined in this config file and
schema.xml.

But what I try to do is to execute this select and update a field of one of
the tables used in the select.

I tried to change the "select" with a "merge", but Oracle can't return
values in a merge so... the select is working, if I try to do " merge into
tablename t (the select) e on ( e.ID = t.ID) when matched then
update set t.timestamp = systimestamp it works (the update) but I don’t
get the data, if I add to this sql, returning e.* I get an error because
returning is not available in Oracle.

So I've a complex select in data-config.xml that works, I can change it with
an update that work but only the update, so I'll not retrieve any data in
the full-import.

So I've no solution at the moment on how to make the select on the database
and then update a value of one of the tables used into this select.

I've tried with UPDATE and not with MERGE but then I got another Oracle
error.

My sql knowledge is limited and I ask in this list searching perhaps for
another kind of solution or idea perhaps into solr config, options etc...


Regards.

-Mensaje original-
De: steve [mailto:sc_shep...@hotmail.com]
Enviado el: martes, 17 de marzo de 2015 10:02
Para: solr-user@lucene.apache.org
Asunto: RE: Select and Update after with DataImportHandler

Hi, maybe I'm missing the point here, but there could be a separate
table/database that has a record inserted after the full import is
completed; this could be part of the same "batch" or script file, or one
that is "chained" after the original query completes.

> From: g...@idieikon.com
> To: solr-user@lucene.apache.org
> Subject: Select and Update after with DataImportHandler
> Date: Tue, 17 Mar 2015 09:47:25 +0100
>
> Hello, we have a sql to retrieve fields from DB and they are indexed in a
full-import defined in a data-config.xml.
>
> It works as expected but now we want to update the database fields so we
know when data was imported.
> So we add to the table being indexed a timestamp. Now we have table1
field1 field2... and fieldN (the timestamp).
>
> Is there any way to indicate "after you finish the select of the
full-import, now you have to do this sql update of the timestamp in the
database table"?
>
> We use Solr 4.6.0
>
>
> We've tried in the same sql, with cursors or transactions, etc, but the
sql select is pretty complex and we can't make it work in such way, but
perhaps the only way to do it is to substitute the select with some kind of
transaction but we don’t know if this is possible to do in the query for the
full-import.
>
> Regards.
>
>
> ---
> El software de antivirus Avast ha analizado este correo electrónico en
busca de virus.
> http://www.avast.com
>
>



---
El software de antivirus Avast ha analizado este correo electrónico en busca de 
virus.
http://www.avast.com

SolrJ insert PDF for different language

[Gabi]

Following the sample in

http://wiki.apache.org/solr/ContentStreamUpdateRequestExample I'm able to
insert a PDF and search words, etc.



ContentStreamUpdateRequest up = new
ContentStreamUpdateRequest("/update/extract");
up.addFile(new File(fileName), "application/pdf");
up.setParam("literal.id", solrId);
up.setParam("uprefix", "attr_");
up.setParam("fmap.content", "attr_content");



My problem is that I have created some fields for a document (in
schema.xml), for example




They use the text_es definition, for spanish text, and if I insert data
directly into this fields it works nice (plural, etc...)

But if I want to add data from PDF (or other) files directly to them, what I
have to change in the code? Can I change the attributes in the
ContentStreamUpdateRequest to do it? What do I have to write in these lines
of code to insert into description the PDF file content?

up.setParam("literal.id", solrId); //I suppose this line does not change
up.setParam("uprefix", "attr_");
up.setParam("fmap.content", "attr_content");

And what I have to add to enter the other fields like put a docName? So when
I search for a word in the document I get "SolrDocument{id= ...,
attr_content=[ ... ], version_=... But I want to add some attributes like
docName or other with my values inserted in my code.

By the way I'm using Solr version 4.6.0 and SolrJ in Java for the code.

Regards.





---
Este mensaje no contiene virus ni malware porque la protección de avast! 
Antivirus está activa.
http://www.avast.com

Upgrading from 4.6 to 7.2.1 SolrJ ContentStreamUpdateRequest problem

[Gabi]

We're trying to upgrade from and old standalone 4.6.0 installation to a
7.2.1 new one.



After some work (of course it's not as easy as install and change schema.xml
and data-config.xml file) it seems to be working with the database but not
with documents.



Database information is loaded initially from a full import from admin web
interface, with a full-import and seems to be ok.

Documents where imported with an application that basically does this
(sample code):




public int indexFile(String fileName, File file, long idDoc, String
province, String province Code) {

HttpSolrClient solr = new HttpSolrClient.Builder(urlString).build();

ContentStreamUpdateRequest up = new
ContentStreamUpdateRequest("/update/extract");

boolean continue = false;

int result = 0;

try {

if(fileName.toLowerCase().endsWith(".pdf")) {

up.addFile(file, "application/pdf");

continue = true;

}

else if (fileName.toLowerCase().endsWith(".docx") ||
fileName.toLowerCase().endsWith(".doc")) {

up.addFile(file, "application/octet-stream");

continue = true;

}

else {

System.err.println("Not supported type!!!");

result = -2;

}

if (continue) {

String aux  = Long.toString(idDoc);

up.setParam("literal.id", "DOC_"+aux);

up.setParam("literal.idDoc", Long.toString(idDoc));

if ( provincia != null)

up.setParam("literal.provinceUser", province);

if ( provinciaCode != null)

up.setParam("literal.provinceCodeUser", provinceCode);

up.setAction(AbstractUpdateRequest.ACTION.COMMIT, true,
true);

solr.request(up);

}

} catch (SolrServerException solr_e) {

solr_e.printStackTrace();

result = -1;

} catch (IOException io_e) {

io_e.printStackTrace();

result = -1;

}

return result;

}



This worked ok on 4.6.0, in the new server we get an exception:

org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://127.0.0.1:8983/solr/collection1: ERROR: [doc=...]
unknown field provincecodeuser

or same error with unknown field 'iddoc'



Error is because it's seraching fot the field with all lower case letters
but in schema.xml and in the code it has some capital letters!!!

up.setParam("literal.idDoc", Long.toString(idDoc));

or  up.setParam("literal.provinceCodeUser", provinceCode);



Why is changing the setParam the param String value to lowercase letters?
this did not happen with 4.6.0 version same application but with 4.6.0 SolrJ
libraries.

How can we use Strings with capital letters? Or is not possible and we have
to change all the aprameter names in the schema.xml and data-config.xml
files?



Regards.













---
El software de antivirus Avast ha analizado este correo electrónico en busca de 
virus.
https://www.avast.com/antivirus