[ANNOUNCE] Apache Solr 6.6.1 released

2017-09-07 Thread Varun Thacker
7 September 2017, Apache Solr™ 6.6.1 available The Lucene PMC is pleased to announce the release of Apache Solr 6.6.1 Solr is the popular, blazing fast, open source NoSQL search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, facete

Re: Solr Commit Thread Blocked because of excessive number of merging threads

2017-09-07 Thread Walter Underwood
Agree, if the merge tuning isn’t working, then stop tuning the merges and go back to defaults. I’ve been running Solr in production for about eight of the last ten years and I’ve never tuned merges. Are your PHP clients sending batches or single documents? 1 k documents per minute seems very, v

Re: Solr Commit Thread Blocked because of excessive number of merging threads

2017-09-07 Thread Erick Erickson
Skimming and to add to what Shawn said about ramBufferSizeMB. It's totally wasted space pretty much since you've set maxDocs to 10,000. It doesn't matter how big ramBufferSizeMB is, when you reach 10,000 docs indexed the buffer will be flushed and set back to zero. And +1 to all of Shawn's commen

Re: Conditions with multiple boosts in bf exists query

2017-09-07 Thread Erick Erickson
I'd sidestep the problem ;) Are these scores 1> known at index time 2> unchanging (at least until the doc is re-indexed)? If so, pre-compute your boost and put it in the doc at index time. The other thing you can do is use payloads to add a float to specific tokens and incorporate them in at

Re: Sort across collapsed document is not working

2017-09-07 Thread Ray Niu
this is not sharded collection, it only had one shard. I want to use collapse to replace current group query, but the result is not same, I feel there are some function issue in collapse plugin 2017-09-07 14:59 GMT-07:00 Erick Erickson : > Is this a sharded collection? group.ngroups isn't support

Re: Solr Commit Thread Blocked because of excessive number of merging threads

2017-09-07 Thread Shawn Heisey
On 9/6/2017 11:54 PM, yasoobhaider wrote: > My team has tasked me with upgrading Solr from the version we are using > (5.4) to the latest stable version 6.6. I am stuck for a few days now on the > indexing part. > > So in total I'm indexing about 2.5million documents. The average document > size is

Re: Consecutive calls to a query give different results

2017-09-07 Thread Erick Erickson
bq: So apparently it IS essential to run optimize after a data load Don't do this if you can avoid it, you run the risk of excessive amounts of your index consisting of deleted documents unless you are following a process whereby you periodically (and I'm talking at least hours, if not once per da

Re: Sort across collapsed document is not working

2017-09-07 Thread Erick Erickson
Is this a sharded collection? group.ngroups isn't supported (see the docs, "group.ngroups and group.facet require that all documents in each group must be co-located on the same shard") in sharded situations so it's not surprising that the results differ. Best, Erick On Thu, Sep 7, 2017 at 10:35

origFreq/freq ratio for filtering spell-check suggestions

2017-09-07 Thread Arnold Bronley
Hi Solr users, I can see there are some parameters that can help in controlling the trigger condition for spellcheck mechanism or filter the spell suggestions like maxQueryFrequency or thresholdTokenFrequency. I could not find a parameter that will filter the suggestions based on (origFreq/freq) r

Re: Solr cloud optimizer

2017-09-07 Thread Tomas Fernandez Lobbe
By default Solr uses the “TieredMergePolicy”[1], but it can be configured in solrconfig, see [2]. Merges can be triggered for different reasons, but most commonly by segment flushes (commits) or other merges finishing. Here is a nice visual demo of segment merging (a bit old but still mostly a

Sort across collapsed document is not working

2017-09-07 Thread Ray Niu
Hello: I tried to use Collapsing Query Parser per following link: https://cwiki.apache.org/confluence/display/solr/Collapse+and+Expand+Results here is the query I am using http:///solr/collection/select?q=groupId:*& fl=id,groupId,date&fq=%7B!collapse%20field=groupId%20sort=%27id%20asc%27%7D& ex

Solr cloud optimizer

2017-09-07 Thread calamita . agostino
Hi  all, I use SolrCloud with  some collections with 3  shards each. Every day I insert and remove documents from collections. I  know that solr starts optimizer in background to optimize indexes. Which  is the policy that solr applies in order  to start optimizer automatically ? Number of del

RE: Customizing JSON response of a query

2017-09-07 Thread Davis, Daniel (NIH/NLM) [C]
Sarvo, I agree with Rick. It is better to put something in front of Solr (or any search engine), because the search engine sort of fits into a 3-tier hierarchy along with the database service: Load Balancer/Httpd front-end > App -> RDBMS Becomes: Load Balancer/Httpd front-end ->

Conditions with multiple boosts in bf exists query

2017-09-07 Thread Eric Kurzenberger
I need to do a bf exists query that matches the following conditions: - IF a_score = 1 AND b_score = 2 THEN boost 30 - IF a_score = 3 AND b_score = 4 THEN boost 20 So far, the bf portion of my query looks like this: if(exists(query({!v="a_score_is:1"})),30,0) But I’m hav

Re: Consecutive calls to a query give different results

2017-09-07 Thread Webster Homer
We have several solr clouds, a couple of them have only 1 replica per shard. We have never observed the problem when we have a single replica only when there are multiple replicas per shard. On Thu, Sep 7, 2017 at 10:08 AM, Webster Homer wrote: > the scores are not the same > Doc > 305340 432.44

Re: Consecutive calls to a query give different results

2017-09-07 Thread Webster Homer
the scores are not the same Doc 305340 432.44238 C2646 428.24185 12837 430.61722 One other thing. I just ran optimize and now document 305340 is consistently the top score. So apparently it IS essential to run optimize after a data load Note we see this behavior fairly commonly on our sol

Re: Consecutive calls to a query give different results

2017-09-07 Thread Webster Homer
the scores are not the same Doc 305340 432.44238 On Thu, Sep 7, 2017 at 10:02 AM, David Hastings < hastings.recurs...@gmail.com> wrote: > "I am concerned that the same > search gives different results after each search. The top document seems to > cycle between 3 different documents" > > > if you

Re: Consecutive calls to a query give different results

2017-09-07 Thread David Hastings
"I am concerned that the same search gives different results after each search. The top document seems to cycle between 3 different documents" if you do debug query on the search, are the scores for the top 3 documents the same or not? you can easily have three documents with the same score, so

Re: Consecutive calls to a query give different results

2017-09-07 Thread Webster Homer
I am not concerned about deleted documents. I am concerned that the same search gives different results after each search. The top document seems to cycle between 3 different documents I have an enhanced collections info api call that calls the core admin api to get the index information for the r

Re: Consecutive calls to a query give different results

2017-09-07 Thread Erick Erickson
Whew! I haven't been lying to people for _years_.. On Thu, Sep 7, 2017 at 5:58 AM, Yonik Seeley wrote: > On Thu, Sep 7, 2017 at 12:47 AM, Erick Erickson > wrote: >> bq: and deleted documents are irrelevant to term statistics... >> >> Did you mean "relevant"? Or do I have to adjust my thinki

Solr Commit Thread Blocked because of excessive number of merging threads

2017-09-07 Thread yasoobhaider
Hi My team has tasked me with upgrading Solr from the version we are using (5.4) to the latest stable version 6.6. I am stuck for a few days now on the indexing part. First I'll list the requirements, then all the configuration settings I have tried. So in total I'm indexing about 2.5million doc

Re: CommitScheduler Thread blocked due to excessive number of Merging Threads

2017-09-07 Thread Shawn Heisey
On 9/7/2017 4:25 AM, yasoobhaider wrote: > So I did a little more digging around why the merging is taking so > long, and it looks like merging postings is the culprit. On the 5.4 > version, merging 500 docs is taking approximately 100 msec, while on > the 6.6 version, it is taking more than 3000 m

Re: deep paging in parallel sql

2017-09-07 Thread Susmit Shukla
you could use filter clause to create a custom cursor since the results are sorted. I had used the approach with raw cloudsolr stream, not with parallelSQL though. This would be useful- https://lucidworks.com/2013/12/12/coming-soon-to-solr-efficient-cursor-based-iteration-of-large-result-sets/ Th

Re: Consecutive calls to a query give different results

2017-09-07 Thread Yonik Seeley
On Thu, Sep 7, 2017 at 12:47 AM, Erick Erickson wrote: > bq: and deleted documents are irrelevant to term statistics... > > Did you mean "relevant"? Or do I have to adjust my thinking _again_? One can make it work either way ;-) Whether a document is marked as deleted or not has no effect on term

RE: [EXTERNAL] - Re: NumberFormatException for multvalue, pint

2017-09-07 Thread Steve Pruitt
Sigh. You are right and thank you for pointing out the obvious, much to my chagrin. :>) Again, thanks. -S -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Wednesday, September 06, 2017 4:42 PM To: solr-user Subject: [EXTERNAL] - Re: NumberFormatException

RE: ERR_SSL_VERSION_OR_CIPHER_MISMATCH

2017-09-07 Thread Younge, Kent A - Norman, OK - Contractor
Still receiving the same issue. I have cloned another machine and it has the same issue. Not sure what to do next. Last resort build machine from scratch and see if it has the same issue if it does then I have no clue what is going on. -Original Message- From: Younge, Kent A

Re: Solr Issue

2017-09-07 Thread Michael Kuhlmann
Hi Patrick, can you attach the query you're sending to Solr and one example result? Or more specific, what are your hl.* parameters? -Michael Am 07.09.2017 um 09:36 schrieb Patrick Fallert: > > Hey Guys,  > i´ve got a problem with my Solr Highlighter.. > When I search for a word, i get some resu

Re: CommitScheduler Thread blocked due to excessive number of Merging Threads

2017-09-07 Thread yasoobhaider
So I did a little more digging around why the merging is taking so long, and it looks like merging postings is the culprit. On the 5.4 version, merging 500 docs is taking approximately 100 msec, while on the 6.6 version, it is taking more than 3000 msec. The difference seems to get worse when more

Solr Issue

2017-09-07 Thread Patrick Fallert
Hey Guys, i´ve got a problem with my Solr Highlighter.. When I search for a word, i get some results. For every result i want to display the highlighted text and here is my problem. Some of the returned documents have a highlighted text the other ones doesnt. I don´t know why it is but i need to