Caching of dynamic external file fields

In Solr there's /ExternalFileFieldReloader/ which is responsible for caching the contents of external files whenever a new searcher is being warmed up. It happens that I've defined a dynamic field to be used as an /ExternalField/ as in /* */ If you have a look inside the code /ExternalFileFiel

RE: External file fields

: Interesting. I will definitely explore this. Just so I'm clear, we can : sort on docValues, but not filter? Is there any situation where external : file fields would work better than docValues? For most field types that support docValues, you can still filter on it even if it'

RE: External file fields

Interesting. I will definitely explore this. Just so I'm clear, we can sort on docValues, but not filter? Is there any situation where external file fields would work better than docValues? -Original Message- From: Chris Hostetter [mailto:hossman_luc...@fucit.org] Sent: F

RE: External file fields

: I did look into updatable docValues, but my understanding is that the : field has to be non-indexed (indexed="false"). I need to be able to sort : on these values. External field fields are sortable. YOu can absolutely sort on a field that is docValues="true" indexed="false" ... that is much

Re: External file fields

ments.html#UpdatingPartsofDocuments-In-PlaceUpdates > > > -Original Message- > From: Erick Erickson [mailto:erickerick...@gmail.com] > Sent: Thursday, February 1, 2018 5:00 PM > To: solr-user > Subject: Re: External file fields > > Have you considered updateabl

RE: External file fields

ments.html#UpdatingPartsofDocuments-In-PlaceUpdates -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Thursday, February 1, 2018 5:00 PM To: solr-user Subject: Re: External file fields Have you considered updateable docValues? Best, Erick On Thu, Feb 1, 2018 at 10:55

Re: External file fields

On 01/02/2018 18:55, Brian Yee wrote: Hello, I want to use external file field to store frequently changing inventory and price data. I got a proof of concept working with a mock text file and this will suit my needs. What is the best way to keep this file updated in a fast way. Ideally I wou

Re: External file fields

Maybe you can try or extend Sematext’s Redis parser: https://github.com/sematext/solr-redis . Downside of this approach is another moving part - Redis. HTH, Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting

Re: External file fields

Have you considered updateable docValues? Best, Erick On Thu, Feb 1, 2018 at 10:55 AM, Brian Yee wrote: > Hello, > > I want to use external file field to store frequently changing inventory and > price data. I got a proof of concept working with a mock text file and this > will suit my needs.

External file fields

Hello, I want to use external file field to store frequently changing inventory and price data. I got a proof of concept working with a mock text file and this will suit my needs. What is the best way to keep this file updated in a fast way. Ideally I would like to read changes from a Kafka qu

Re: Real Time Search and External File Fields

Thanks for the replies. I made the changes so that the external file field is loaded per:

Re: Real Time Search and External File Fields

On 10/8/2016 1:18 PM, Mike Lissner wrote: > I want to make sure I understand this properly and document this for > futurepeople that may find this thread. Here's what I interpret your > advice to be: > 0. Slacken my auto soft commit interval to something more like a minute. Yes, I would do this.

Re: Real Time Search and External File Fields

I chose 16 as a place to start. You usually reach diminishing returns pretty quickly, i feel it's a mistake to set your autowarm counts to, say 256 (and I've seen this in the thousands) unless you have some proof that it's useful to bump higher. But certainly if you set them to 16 and see spikes j

Re: Real Time Search and External File Fields

With time-oriented data, you can use an old trick (goes back to Infoseek in 1995). Make a “today” collection that is very fresh. Nightly, migrate new documents to the “not today” collection. The today collection will be small and can be updated quickly. The archive collection will be large and

Re: Real Time Search and External File Fields

On Fri, Oct 7, 2016 at 8:18 PM Erick Erickson wrote: > What you haven't mentioned is how often you add new docs. Is it once a > day? Steadily > from 8:00 to 17:00? > Alas, it's a steady trickle during business hours. We're ingesting court documents as they're posted on court websites, then sendi

Re: Real Time Search and External File Fields

On Sat, Oct 8, 2016 at 8:46 AM Shawn Heisey wrote: > Most soft commit > > documentation talks about setting up soft commits with of > about a > > second. > > IMHO any documentation that recommends autoSoftCommit with a maxTime of > one second is bad documentation, and needs to be fixed. Where h

Re: Real Time Search and External File Fields

On 10/7/2016 6:19 PM, Mike Lissner wrote: > Soft commits seem to be exactly the thing for this, but whenever I open a > new searcher (which soft commits seem to do), the external file is > reloaded, and all queries are halted until it finishes loading. When I just > measured, this took about 30 sec

Re: Real Time Search and External File Fields

bq: Most soft commit documentation talks about setting up soft commits with of about a second. I think this is really a consequence of this being included in the example configs for illustrative purposes, personally I never liked this. There is no one right answer. I've seen soft commit interval

Real Time Search and External File Fields

I have an index of about 4M documents with an external file field configured to do boosting based on pagerank scores of each document. The pagerank file is about 93MB as of today -- it's pretty big. Each day, I add about 1,000 new documents to the index, and I need them to be available as soon as

Re: Question about external file fields

I guess you refer to this post? http://1opensourcelover.wordpress.com/2013/07/02/solr-external-file-fields/ If so .. he already provides at least one possible use case: *snip* We use Solr to serve our company’s browse pages. Our browse pages are similar to how a typical Stackoverflow tag page

Question about external file fields

Hi, I read this post http://1opensourcelover.wordpress.com/ about EEF's and I found very interesting. Can someone give me more use cases about the utility of EEF's? /Yago - Best regards -- View this message in context: http://lucene.472066.n3.nabble.com/Question-about-ext

Re: Replicating files containing external file fields

nal_1 > > Tor each file. > > (This is what Erick was referring to.) > > Sorry, I don't have the answer to the reload question at the tip of my > tongue. > > > -- Jack Krupansky > > -Original Message- From: Arun Rangarajan > Sent: Friday, June 28,

Re: Replicating files containing external file fields

to.) Sorry, I don't have the answer to the reload question at the tip of my tongue. -- Jack Krupansky -Original Message- From: Arun Rangarajan Sent: Friday, June 28, 2013 7:42 PM To: solr-user@lucene.apache.org Subject: Re: Replicating files containing external file fields Ja

Re: Replicating files containing external file fields

}/replication 00:01:00 The confFiles are under the dir: /var/solr/application-cores/List/conf and the external file fields are like: /var/solr-data/List/external_* Should I add /var/solr-data/List/external_* to confFiles like this? solrconfig.xml,data-config.xml,schema.xml,stopwords.txt,synonyms.txt

Re: Replicating files containing external file fields

Show us your directive. Maybe there is some subtle error in the file name. -- Jack Krupansky -Original Message- From: Arun Rangarajan Sent: Friday, June 28, 2013 1:06 PM To: solr-user@lucene.apache.org Subject: Re: Replicating files containing external file fields Erick, Thx for

Re: Replicating files containing external file fields

rReplication I understand that > index > > dir and any files under the conf dir can be replicated to slaves. I want > to > > know if there is any way the files under the data dir containing external > > file fields can be replicated. These are not replicated by default. >

Re: Replicating files containing external file fields

ki.apache.org/solr/SolrReplication I understand that index > dir and any files under the conf dir can be replicated to slaves. I want to > know if there is any way the files under the data dir containing external > file fields can be replicated. These are not replicated by default. > Current

Replicating files containing external file fields

>From https://wiki.apache.org/solr/SolrReplication I understand that index dir and any files under the conf dir can be replicated to slaves. I want to know if there is any way the files under the data dir containing external file fields can be replicated. These are not replicated by defa