bq: I wonder if it won’t be simpler for me to write a custom handler Probably not, that would be Java too ;)...
OK, back up a bit. You can change your schema such that the full-text field _is_ stored, I don't quite know what the default field is from memory, but you must be searching against it ;). It sounds like you're using the defaults and it's _probably_ _text_. And my guess is that you're searching on that field even though you don't specify, see the "df" entry in your solrconfig.xml file. There's no reason you can't change that to stored="true" (reindex of course). Nothing that you've mentioned so far looks like it should take anything except getting your configurations to be what you need, so don't make more work for yourself than you need to ;). After that, see the link Shawn provided... Best, Erick On Wed, Mar 8, 2017 at 8:22 AM, Nicolas Bouillon <nico2000...@yahoo.com.invalid> wrote: > Hi Erick > > Thanks a lot for the elaborated answer. Let me give some precisions: > > 1. I upload the docs using an AJAX post multiform to my server. > 2. The PHP target of the post, takes the file and stores it on disk > 3. If the file is moved successfully from TEMP files to final destination, I > then call SOLR as follows: > > It’s a curl POST request: > > URL: http://my_server:8983/solr/my_core/update/extract/?"; . $fields . > "&literal.id=" . $id . "&filetypes=*&commit=true > HEADER: Content-type: multipart/form-data > POSTFIELDS: the entire file that has just been stored > (BTW, it’s PHP specific but I send a CurlFile in an array as follows: > array('myfile' => $cfile) > > In the URL, the parameter $fields contains the following: > > $fields = "literal.kref=" . $id . "&literal.ktype=" . $type . > "&literal.kattachment=" . $attachment; > > Where kref, ktype and kattachment are my custom fields (that I added to the > schema.xml previously) > > So, indeed it’s Tika that extracts the info. I didn’t change anything to the > ExtractHandler. > > I read about the fact that all fields must be marked as stored=true but: > > - I checked in the schema, all the fields that matter (Tika default extracted > fields) and my customer fields are stored=true. > - I suppose that the full-text index is not stored in a field? And therefore > cannot be marked as stored? > > I manage to upload files and mark my docs with metadata but I have existing > files where I would like to update my fields (kref, …) without re-extracting > and I’d like also to allow for re-indexing if needed without overriding my > fields. > > I’m stuck… I wonder if it won’t be simpler for me to write a custom handler > of some sort but I don’t really program in Java. > > Cheers > > Nico > >> On 8 Mar 2017, at 17:03, Erick Erickson <erickerick...@gmail.com> wrote: >> >> Nico: >> >> This is the place for such questions! I'm not quite sure the source >> of the docs. When you say you "extract", does that mean you're using >> the ExtractingRequestHandler, i.e. uploading PDF or Word etc. to Solr >> and letting Tika parse it out? IOW, where is the fulltext coming from? >> >> For adding tags any time, Solr has "Atomic Updates" that has a couple >> of requirements, mainly you have to set stored="true" for all your >> fields _except_ the destinations for any <copyField> directives. Under >> the covers this pulls the stored data from Solr, overlays it with the >> new data you've sent and re-indexes it. The expense here is that your >> index will increase in size, but storing the data doesn't mean much of >> an increase in JVM requirements. That is, say your index doubles in >> size. Your JVM heap requirements may increase 5% (and, frankly I doubt >> that much, but I've never measured). FWIW, the on-disk size should >> increase by roughly 50% of the raw data size. WARNING: "raw data size" >> is the size _after_ extraction, so say you're indexing a 1K XML doc >> where the tags are taking up .75K. Then the on-disk memory should go >> up roughly .125K (50% of .25K).. >> >> Don't worry about "thousands" of docs ;) On my laptop I index over 1K >> Wikipedia articles a second (YMMV of course). Without any particular >> tuning. Without sharding. Very often the most expensive part of >> indexing is acquiring the data in the first place, i.e. getting it >> from a DB or extracting it from Tika. Solr will handle quite a load. >> >> And, if you're using the ExtractingRequestHandler, I'd seriously think >> about moving it to a Client. Here's a Java example: >> https://lucidworks.com/2012/02/14/indexing-with-solrj/ >> >> Best, >> Erick >> >> On Wed, Mar 8, 2017 at 7:46 AM, Nicolas Bouillon >> <nico2000...@yahoo.com.invalid> wrote: >>> Dear SOLR friends, >>> >>> I developed a small ERP. I produce PDF documents linked to objects in my >>> ERP: invoices, timesheets, contracts, etc... >>> I have also the possibility to attach documents to a particular object and >>> when I view an invoice for instance, I can see the attached documents. >>> >>> Until now, I was adding reference to these documents in my DB and store >>> docs on the server. >>> Still, I found it cumbersome and not flexible enough, so I removed the >>> table documents from my DB and decided to use SOLR to add metadata to the >>> documents in the index. >>> >>> Currently, I have the following custom fields: >>> - ktype (string): invoice, contract, etc… >>> - kattachment (int): 0 or 1 >>> - kref (int): reference in DB of linked object, ex: 10 (for contract 10 in >>> DB) >>> - ktags (strings, mutifield): free tags, ex: customerX, consulting, >>> development >>> >>> Each time I upload a document, I store in on server and then add it to SOLR >>> using "extract" adding the metadata at the same time. It works fine. >>> >>> I would like now 3 things: >>> >>> - For existing documents that have not been extracted with metadata >>> altogether at upload (documents uploaded before I developed the >>> functionality), I'd like to update them with the proper metadata without >>> losing the full-text search >>> - Be able to add anytime tags to the ktags field after upload whilst >>> keeping full-text search >>> - In case I have to re-index, I want to be sure I don't have to restart >>> everything from scratch. >>> In a few months, I expect to have thousands of docs in my >>> system....and then I'll add emails >>> >>> I have very little experience in SOLR. I know I can re-perform an extract >>> instead of an update when I modify a field but I'm pretty sure it's not the >>> right thing to do + performance problems can arise. >>> >>> What do you suggest me to do? >>> >>> I thought about storing the metadata linked to each document separately (in >>> DB or separate XML file individually or one XML for all) but I'm pretty >>> sure it will be very slow after a while. >>> >>> Thx a lot in advance fro your precious help. >>> This is my first message to the user list, please excuse anything I may >>> have done wrong…I learn fast, don’t worry.. >>> >>> Regards >>> >>> Nico >>> >>> My configuration: >>> >>> Synology 1511 running DSM 6.1 >>> Docker container for SOLR using latest stable version >>> 1 core called “katalyst” containing index of all documents >>> >>> ERP is written in PHP/Mysql for backend and Jquery/Bootstrap for front-end >>> >>> I have a test env on OSX Sierra running docker, a prod environment on >>> Synology >>> >>> >
