Solr Replication being flaky (6.2.0)

2018-01-19 Thread Pouliot, Scott
So we're running Solr in a Master/Slave configuration (1 of each) and it seems that the replication stalls or stops functioning every now and again. If we restart the Solr service or optimize the core it seems to kick back in again. Anyone have any idea what might be causing this? We do have a

Re: Solr Replication being flaky (6.2.0)

2018-01-19 Thread David Hastings
This happens to me quite often as well. Generally on the replication admin screen it will say its downloading a file, but be at 0 or a VERY small kb/sec. Then after a restart of the slave its back to downloading at 30 to 100 mg/sec. Would be curious if there actually is a solution to this aside

RE: Solr Replication being flaky (6.2.0)

2018-01-19 Thread Pouliot, Scott
I'm at the point now where I may end up writing a script to compare master/slave nightly...and trigger an optimize or solr restart if there are any differences. Of course I have to check 150+ cores...but it could be done. I'm just hoping I don't need to go that route -Original Message

Re: Solr Replication being flaky (6.2.0)

2018-01-19 Thread Shawn Heisey
On 1/19/2018 7:50 AM, Pouliot, Scott wrote: So we're running Solr in a Master/Slave configuration (1 of each) and it seems that the replication stalls or stops functioning every now and again. If we restart the Solr service or optimize the core it seems to kick back in again. Anyone have any

RE: Solr Replication being flaky (6.2.0)

2018-01-19 Thread Pouliot, Scott
I do have a ticket in with our systems team to up the file handlers since I am seeing the "Too many files open" error on occasion on our prod servers. Is this the setting you're referring to? Found we were set to to 1024 using the "Ulimit" command. -Original Message- From: Shawn Heise

Re: Solr Replication being flaky (6.2.0)

2018-01-19 Thread Erick Erickson
Scott: We usually recommend setting files and processes very, very high. Like 65K high. Or unlimited if you can. Plus max user processes should also be bumped very high as well, like 65K as well. Plus max memory and virtual memory should be unlimited. We've included warnings at startup for open

RE: Solr Replication being flaky (6.2.0)

2018-01-19 Thread Pouliot, Scott
Erick, Thanks! Could these settings be toying with replication? Solr itself seems to be working like a champ, except when things get out of sync. Scott -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Friday, January 19, 2018 12:27 PM To: solr-user Subj

Re: Solr Replication being flaky (6.2.0)

2018-01-19 Thread Erick Erickson
"Could be", certainly. "Definitely is" is iffier ;)... But the statement "If we restart the Solr service or optimize the core it seems to kick back in again.", especially the "optimize" bit (which, by the way you should do only if you have the capability of doing it periodically [1]) is some evide

RE: Solr Replication being flaky (6.2.0)

2018-01-19 Thread Pouliot, Scott
That's evidence enough for me to beat on our systems guys to get these file handles upped and cross my fingers then! -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Friday, January 19, 2018 1:18 PM To: solr-user Subject: Re: Solr Replication being flaky (6

Re: Solr Replication being flaky (6.2.0)

2018-01-19 Thread Shawn Heisey
On 1/19/2018 8:54 AM, Pouliot, Scott wrote: I do have a ticket in with our systems team to up the file handlers since I am seeing the "Too many files open" error on occasion on our prod servers. Is this the setting you're referring to? Found we were set to to 1024 using the "Ulimit" command.

Re: Solr Replication being flaky (6.2.0)

2018-01-19 Thread Shawn Heisey
On 1/19/2018 11:27 AM, Shawn Heisey wrote: On 1/19/2018 8:54 AM, Pouliot, Scott wrote: I do have a ticket in with our systems team to up the file handlers since I am seeing the "Too many files open" error on occasion on our prod servers.  Is this the setting you're referring to?  Found we were

Issue with solr.HTMLStripCharFilterFactory

2018-01-19 Thread Fiz Ahmed
Hi Solr Experts, I am using the HTMLStripCharFilterFactory for removing tags in Body element. Body contains data like Ipad I made changes in managed schema . ---

RE: Solr Replication being flaky (6.2.0)

2018-01-19 Thread Pouliot, Scott
Working on that now to see if it helps us out. Solr process is NOT dying at all. Searches are still working as expected, but since we load balance requestsif the master/slave are out of sync the search results vary. The advice is MUCH appreciated! -Original Message- From: Shawn

Re: Adding a child doc incrementally

2018-01-19 Thread S G
Restriction to a single shard seems like a big limitation for us. Also, I was hoping that this was something Solr provided out of the box. (Like https://lucene.apache.org/solr/guide/6_6/updating-parts-of-documents.html#UpdatingPartsofDocuments-In-PlaceUpdates ) Something like: { "id":"parents-id

Strange Alias behavior

2018-01-19 Thread Webster Homer
I just discovered some odd behavior with aliases. We are in the process of converting over to use aliases in solrcloud. We have a number of collections that applications have referenced the collections from when we used standalone solr. So we created alias names to match the name that the java app

Re: Preserve order during indexing

2018-01-19 Thread Webster Homer
db order isn't generally defined, unless you are using an explicit "order by" on your select. Default behavior would vary by database type and even release of the database. You can index the fields that you would "order by" in the db, and sort on those fields in solr On Thu, Jan 18, 2018 at 10:17

Re: SOLR Data Backup

2018-01-19 Thread S G
Another option is to have CDCR enabled for Solr and replicate your data to another Solr cluster continuously. BTW, why do we not recommend having Solr as a source of truth? On Thu, Jan 18, 2018 at 4:08 AM, Florian Gleixner wrote: > Am 18.01.2018 um 10:21 schrieb Wael Kader: > > Hello, > > > > W

Re: Strange Alias behavior

2018-01-19 Thread Wenjie Zhang (Jack)
Why would you create an alias with an existing collection name? Sent from my iPhone > On Jan 19, 2018, at 14:14, Webster Homer wrote: > > I just discovered some odd behavior with aliases. > > We are in the process of converting over to use aliases in solrcloud. We > have a number of collection

Re: Strange Alias behavior

2018-01-19 Thread Webster Homer
I created the alias with an existing collection name because our code base which was created with stand alone solr was a pain to change. I did test that the alias took precedence over the collection, when I did a search. On Fri, Jan 19, 2018 at 4:22 PM, Wenjie Zhang (Jack) < wenjiezhang2...@gmail.

Re: Strange Alias behavior

2018-01-19 Thread Shawn Heisey
On 1/19/2018 3:53 PM, Webster Homer wrote: I created the alias with an existing collection name because our code base which was created with stand alone solr was a pain to change. I did test that the alias took precedence over the collection, when I did a search. The ability to create aliases a

Re: Issue with solr.HTMLStripCharFilterFactory

2018-01-19 Thread Shawn Heisey
On 1/19/2018 11:56 AM, Fiz Ahmed wrote: But When I Query in Solr Admin.. I am still getting the Search results with Html Tags in it. Search results will always contain the actual content that was indexed. Analysis only happens to indexed data and/or queries, not stored data. This is how Solr

Re: Strange Alias behavior

2018-01-19 Thread Webster Homer
It seems like a useful feature, especially for migrating from standalone to solrcloud, at least if the precedence of alias to collection is defined and enforced. On Fri, Jan 19, 2018 at 5:01 PM, Shawn Heisey wrote: > On 1/19/2018 3:53 PM, Webster Homer wrote: > >> I created the alias with an exi

BinaryResponseWriter fetches unnecessary fields?

2018-01-19 Thread Wei
Hi all, We observe that solr query time increases significantly with the number of rows requested, even all we retrieve for each document is just fl=id,score. Debugged a bit and see that most of the increased time was spent in BinaryResponseWriter, converting lucene document into SolrDocument.