Re: Scaling issue with Solr

Thanks Eric and others for the replies. In the prototype we are ingesting only 300 gb of data into the 2 node solr cloud. I am well aware of the fact that for 4Tb we need much larger setup. We came across some posts suggesting keeping the hard commit intervals as small as possible in case if t

Re: fq: OR operator (sometimes) not working

Thanks Erick for pushing me into the right direction. so sow=false, but i think that it is the default behavior so I didn't expect this to cause any strange outcome. However the reason Folder_id is being treated differently than the others is the schema definition. Folder_id is a long. While file

Re: fq: OR operator (sometimes) not working

OK, that's definitely weird. A separate fq clause like fq={!q.op=OR}file_type:(jpg jpeg) should _not_ parse in to: file_type:jpg jpeg Hmmm, any possibility that Split On Whitespace is somehow being set (SOW) to false? Why in the world it would only show up like this is a mystery, just askin'. It

Re: fq: OR operator (sometimes) not working

Thanks Erik. Yes some similar queries are also working for me. "file_type:(jpg%20OR%20jpeg)" and "{!q.op=OR}file_type:(jpg OR jpeg)" are translated into the following which is correct. - "file_type:jpg file_type:jpeg" While "{!q.op=OR}file_type:(jpg jpeg)" is translated into file_type:jpg jp

Re: Scaling issue with Solr

NOTE: you'll also have to change your hard commit options to be long enough to make the ram buffer fill up. Yes, recoveries will likely be longer in the event of a crash. Frankly at your ingestion rate though you'll be having "fun" with this in the event of a crash. But as others have noted, your

Re: fq: OR operator (sometimes) not working

1> similar queries work for me just fine with the techproducts exapmle 2> that's not what I wanted, you just reiterated the _input_. I asked for the results when adding &debug=query to the string so you can see the parsed query. You should see something similar to: "parsed_filter_queries":["file_t

Re: Scaling issue with Solr

You may find that buying some more memory will be your best bang for the buck in your set up. 32-64 gb isn’t expensive, > On Dec 27, 2017, at 6:57 PM, Suresh Pendap wrote: > > What is the downside of configuring ramBufferSizeMB to be equal to 5GB ? > Is it only that the window of time for flu

Re: fq: OR operator (sometimes) not working

1. input: fq={!q.op=OR}file_type:(jpg%20jpeg) (fails, no results) - fq: [ - "id:file_258470818866", - "{!q.op=OR}file_type:(jpg jpeg)" ], 2. input: fq={!q.op=OR}file_type:(jpg%20OR%20jpeg) (This works) - fq: [ - "id:file_258470818866", - "{!q.op=OR}file_t

Re: Scaling issue with Solr

What is the downside of configuring ramBufferSizeMB to be equal to 5GB ? Is it only that the window of time for flush is larger, so recovery time will be higher in case of a crash? Thanks Suresh On 12/27/17, 1:34 PM, "Erick Erickson" wrote: You are probably hitting more and more background

Re: fq: OR operator (sometimes) not working

What does adding &debug=query show in the two cases? Best, Erick On Wed, Dec 27, 2017 at 3:40 PM, Nawab Zada Asad Iqbal wrote: > Hi, > > Are the following two queries equal: > > In my understanding, I can specify the arguments the operator once in the > {} local parameter syntax (example 1) or I

fq: OR operator (sometimes) not working

Hi, Are the following two queries equal: In my understanding, I can specify the arguments the operator once in the {} local parameter syntax (example 1) or I can interleave OR between different clauses (example 2). But I am getting my result in the second case only. What am I doing wrong? This

Re: Scaling issue with Solr

You seem to have the soft and hard commits the wrong way around. Hard commit is more expensive. On 28 December 2017 at 09:10, Walter Underwood wrote: > Why are you using Solr for log search? Elasticsearch is widely used for > log search and has the best infrastructure for that. > > For the past

Re: Scaling issue with Solr

Why are you using Solr for log search? Elasticsearch is widely used for log search and has the best infrastructure for that. For the past few years, it looks like a natural market segmentation is happening, with Solr used for product search and ES for log search. By now, I would not expect Solr

Re: Scaling issue with Solr

You are probably hitting more and more background merging which will slow things down. Your system looks to be severely undersized for this scale. One thing you can try (and I emphasize I haven't prototyped this) is to increase your RamBufferSizeMB solrcofnig.xml setting significantly. By default,

SOLR 7.2 and LTR

Hi, I am using SOLR 7.0 and use the ltr parser. The configuration I use works nicely under SOLR 7.0.0. I am trying to upgrade to 7.2.0 but whenever I want to use my handler, I get an exception: "rq parameter must be a RankQuery" The exact response is: org.apache.solr.common.SolrException org.ap

Using _default configset in standalone mode

Hello, After reading this docs, I'm trying to achieve the following with version 7.2.0: - When creating a new collection, if you *do not specify a configSet*, the _default will be used. -

Enable default wildcard search

Hi everybody! I try integration Solr 6.6.1 with my email server (dovecot 2.32). I have the following settings: schema.xml - https://pastebin.com/1XXWTs8V solrconfig.xml - https://pastebin.com/5HSswCcv But under these settings, the search works only on the full coincidence, for instance, if I s

Scaling issue with Solr

Hello All, We have been building a Solr based solution to hold a large amount of data (approx 4 TB/day or > 24 Billion documents per day). We are developing a prototype on a small scale just to evaluate Solr performance gradually. Here is our setup configuration. Solr cloud: node1: 16 GB RAM,

7.1.0 weird messages bad core before recovery

Hello, I just had a bad core that needed recovery after restart, first it told me this: org.apache.solr.common.SolrException: Unable to locate core logs_shard1_replica1 at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$5(CoreAdminOperation.java:150) at org.apache

Sharding and Replication

Hello All, We had two system where we were doing Master Slave Replication, we used to do delta-import every 24 hours since we did not want the near real-time data. Now since our data is increasing we thought of adding one more machine to the master slave replication. I thought of creating a Sh