Re: Design Question

[Emir Arnautović]

Hi Deepthi,
I never tried case with multivalue field.
One note about join - your dictionary index must be single shard index and must 
be replicated on all Solr nodes where your other index is.

Let us know how it worked out for you.

Thanks,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 8 Feb 2018, at 14:59, Deepthi P  wrote:
> 
> Hi Emir,
> 
> Thank you. The dictionary is static but the descriptions are long text. If
> i denormalize each document will have  100+ descriptions and I have 8
> million records in collection. There is lot of repetition of descriptions
> and the index becomes large. I am trying to avoid that. Also data load time
> adds to it as I have to iterate over each ID and add description.
> 
> Will join work if ID is multivalued field in Solr collection and single
> value field in dictionary ? Please advise.
> 
> Thanks
> Deepthi
> 
> On Thu, Feb 8, 2018 at 2:55 AM, Emir Arnautović <
> emir.arnauto...@sematext.com> wrote:
> 
>> Hi Deepthi,
>> Is dictionary static? Can value for some id change? If static, and if
>> query performance matters to you, the best and also the simplest solution
>> is to denormalise data and store dictionary values with docs.
>> 
>> Alternative is to use join query parser: https://lucene.apache.org/
>> solr/guide/6_6/other-parsers.html#OtherParsers-JoiningAcrossCollections <
>> https://lucene.apache.org/solr/guide/6_6/other-parsers.html#OtherParsers-
>> JoiningAcrossCollections>
>> 
>> HTH,
>> Emir
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>> 
>> 
>> 
>>> On 8 Feb 2018, at 07:21, Deepthi P  wrote:
>>> 
>>> I have a dictionary of 2 ID's and their description which is in a
>>> collection. There is another solr collection in which each document have
>> 10
>>> or more ID's(multi valued field). I would like to text search in the
>>> dictionary and bring back the matched ID's and search these ID's in solr
>>> collection. For example if I search for 'fox' in the dictionary
>> collection
>>> and it returns 40 ID's, I would like to sent these 40 ID's to solr
>>> collection and bring back documents that have these ID's.Please suggest
>> the
>>> best way to implement this.
>> 
>> 

Benchmarking Solr Query performance

[Diego Ceccarelli (BLOOMBERG/ LONDON)]

Hi all, 
We would like to perform a benchmark of 
https://issues.apache.org/jira/browse/SOLR-11831
The patch improves the performance of grouped queries asking only for one 
result per group (aka. group.limit=1).

I remember seeing a page showing a benchmark of the query performance on 
Wikipedia, 
Do you know if there is a way in solr to reproduce the same benchmark? Or some 
independent library to do that? 

thanks,
Diego

solrcloud Auto-commit doesn't seem reliable

[Webster Homer]

I have observed this behavior with several versions of solr (4.10, 6.1, and
now 7.2)

I look in the admin console and look at a core and see that it is not
"current"
I also notice that there are lots of segments etc...

I look at the latest timestamp on a record in the collection and see that
it is over 24 hours old.

I send a commit to the collection, and then see that the core is now
current, and the segments are fewer. The commit worked


This is the setting in solrconfig.xml
 ${solr.autoCommit.maxTime:6} <
openSearcher>false 

Our Solr startup sets solr.autoCommit
-Dsolr.autoCommit.maxTime=6

Yet it looks like we don't get commits regularity. This morning I saw
several collections that hadn't had a hard commit in more than 24 hours.

How can this be? I don't feel that we can rely on Solr's autocommit
capability and this is disturbing

-- 


This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, 
you must not copy this message or attachment or disclose the contents to 
any other person. If you have received this transmission in error, please 
notify the sender immediately and delete the message and any attachment 
from your system. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not accept liability for any omissions or errors in this 
message which may arise as a result of E-Mail-transmission or for damages 
resulting from any unauthorized changes of the content of this message and 
any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not guarantee that this message is free of viruses and does 
not accept liability for any damages caused by any virus transmitted 
therewith.

Click http://www.emdgroup.com/disclaimer to access the German, French, 
Spanish and Portuguese versions of this disclaimer.

Re: solrcloud Auto-commit doesn't seem reliable

[Shawn Heisey]

On 2/9/2018 8:44 AM, Webster Homer wrote:

I look at the latest timestamp on a record in the collection and see that
it is over 24 hours old.

I send a commit to the collection, and then see that the core is now
current, and the segments are fewer. The commit worked

This is the setting in solrconfig.xml
 ${solr.autoCommit.maxTime:6} <
openSearcher>false 


As recommended, you have openSearcher set to false.

This means that these commits are NEVER going to make changes visible.

Don't go and change openSearcher to true.  It is STRONGLY recommended to 
have openSearcher=false in your autoCommit settings.  The reason for 
this configuration is that it prevents the transaction log from growing 
out of control.  With openSearcher=false, those commits will be very 
fast.  This is because it's opening the searcher that's slow, not the 
process of writing data to disk.


Here's the recommended reading on the subject:

https://lucidworks.com/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

For change visibility, configure autoSoftCommit, probably with a 
different interval than you have for autoCommit.  I would recommend a 
longer interval.  Or include the commitWithin parameter on at least some 
of your update requests.  Or send explicit commit requests, preferably 
as soft commits.


Thanks,
Shawn

Re: solrcloud Auto-commit doesn't seem reliable

[Webster Homer]

I we do have autoSoftcommit set to 3 seconds. It is NOT the visibility of
the records that is my primary concern. I am concerned about is the
accumulation of uncommitted tlog files and the larger number of deleted
documents.

I am VERY familiar with the Solr documentation on this.

On Fri, Feb 9, 2018 at 10:08 AM, Shawn Heisey  wrote:

> On 2/9/2018 8:44 AM, Webster Homer wrote:
>
>> I look at the latest timestamp on a record in the collection and see that
>> it is over 24 hours old.
>>
>> I send a commit to the collection, and then see that the core is now
>> current, and the segments are fewer. The commit worked
>>
>> This is the setting in solrconfig.xml
>>  ${solr.autoCommit.maxTime:6} <
>> openSearcher>false 
>>
>
> As recommended, you have openSearcher set to false.
>
> This means that these commits are NEVER going to make changes visible.
>
> Don't go and change openSearcher to true.  It is STRONGLY recommended to
> have openSearcher=false in your autoCommit settings.  The reason for this
> configuration is that it prevents the transaction log from growing out of
> control.  With openSearcher=false, those commits will be very fast.  This
> is because it's opening the searcher that's slow, not the process of
> writing data to disk.
>
> Here's the recommended reading on the subject:
>
> https://lucidworks.com/understanding-transaction-logs-
> softcommit-and-commit-in-sorlcloud/
>
> For change visibility, configure autoSoftCommit, probably with a different
> interval than you have for autoCommit.  I would recommend a longer
> interval.  Or include the commitWithin parameter on at least some of your
> update requests.  Or send explicit commit requests, preferably as soft
> commits.
>
> Thanks,
> Shawn
>

-- 


This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, 
you must not copy this message or attachment or disclose the contents to 
any other person. If you have received this transmission in error, please 
notify the sender immediately and delete the message and any attachment 
from your system. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not accept liability for any omissions or errors in this 
message which may arise as a result of E-Mail-transmission or for damages 
resulting from any unauthorized changes of the content of this message and 
any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not guarantee that this message is free of viruses and does 
not accept liability for any damages caused by any virus transmitted 
therewith.

Click http://www.emdgroup.com/disclaimer to access the German, French, 
Spanish and Portuguese versions of this disclaimer.

Re: solrcloud Auto-commit doesn't seem reliable

[Webster Homer]

A little more background. Our production Solrclouds are populated via CDCR,
CDCR does not replicate commits, Commits to the target clouds happen via
autoCommit settings

We see relvancy scores get inconsistent when there are too many deletes
which seems to happen when hard commits don't happen.

On Fri, Feb 9, 2018 at 10:25 AM, Webster Homer 
wrote:

> I we do have autoSoftcommit set to 3 seconds. It is NOT the visibility of
> the records that is my primary concern. I am concerned about is the
> accumulation of uncommitted tlog files and the larger number of deleted
> documents.
>
> I am VERY familiar with the Solr documentation on this.
>
> On Fri, Feb 9, 2018 at 10:08 AM, Shawn Heisey  wrote:
>
>> On 2/9/2018 8:44 AM, Webster Homer wrote:
>>
>>> I look at the latest timestamp on a record in the collection and see that
>>> it is over 24 hours old.
>>>
>>> I send a commit to the collection, and then see that the core is now
>>> current, and the segments are fewer. The commit worked
>>>
>>> This is the setting in solrconfig.xml
>>>  ${solr.autoCommit.maxTime:6} <
>>> openSearcher>false 
>>>
>>
>> As recommended, you have openSearcher set to false.
>>
>> This means that these commits are NEVER going to make changes visible.
>>
>> Don't go and change openSearcher to true.  It is STRONGLY recommended to
>> have openSearcher=false in your autoCommit settings.  The reason for this
>> configuration is that it prevents the transaction log from growing out of
>> control.  With openSearcher=false, those commits will be very fast.  This
>> is because it's opening the searcher that's slow, not the process of
>> writing data to disk.
>>
>> Here's the recommended reading on the subject:
>>
>> https://lucidworks.com/understanding-transaction-logs-softco
>> mmit-and-commit-in-sorlcloud/
>>
>> For change visibility, configure autoSoftCommit, probably with a
>> different interval than you have for autoCommit.  I would recommend a
>> longer interval.  Or include the commitWithin parameter on at least some of
>> your update requests.  Or send explicit commit requests, preferably as soft
>> commits.
>>
>> Thanks,
>> Shawn
>>
>
>

-- 


This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, 
you must not copy this message or attachment or disclose the contents to 
any other person. If you have received this transmission in error, please 
notify the sender immediately and delete the message and any attachment 
from your system. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not accept liability for any omissions or errors in this 
message which may arise as a result of E-Mail-transmission or for damages 
resulting from any unauthorized changes of the content of this message and 
any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not guarantee that this message is free of viruses and does 
not accept liability for any damages caused by any virus transmitted 
therewith.

Click http://www.emdgroup.com/disclaimer to access the German, French, 
Spanish and Portuguese versions of this disclaimer.

Re: Querying on sum of child documents

[Prath]

We need  sum of experiences from each child documents. Using this expression
"exp:[4 TO 7]" we are directly querying the single child document with the
experience between 4 to 7 but not sum of experiences of all child
documents.For that we have to do something. Could you please help me out
with that logic.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: High CPU and Physical Memory Usage in solr with 4000 user load

[rubi.hali]

Hi Shawn

We tried one more round of testing after We increased CPU cores on each
server from 4 to 16 which amounts to 64 cores in total against 4 slaves.But
Still CPU Usage was higher.So We took the thread dumps on one of our slaves
and found that threads were blocked. Have also attached them.

As far as document size is considered, we have only 1.5 GB index(only one)
size amounting to 4 lakh docs  on each server and query load in span of 10
minutes on each of slaves was distributed as
slave 1- 1258
slave 2 - 512
slave 3 - 256
slave 4 - 1384

We are using Linux OS. 

As threads are blocked , our cpu is reaching to 90% even even when heap or
os memory is not getting used at all.

One of the BLOCKED Thread Snippet
Most of the threads are blocked either on Jetty Connector or FieldCacheIMPL
for DOC values.  Slave1ThreadDump5.txt
  
Slave1ThreadDump4.txt
  

"qtp1205044462-26-acceptor-1@706f9d47-ServerConnector@7ea87d3a{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}"
#26 prio=5 os_prio=0 tid=0x7f2530501800 nid=0x4f9c waiting for monitor
entry [0x7f2505eb8000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:233)
- waiting to lock <0x00064067fc68> (a java.lang.Object)
at
org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:373)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Using Synonyms as a feature with LTR

[Roopa Rao]

I am trying to use synonyms expansion as a feature in LTR

Any input on a feature using synonym expansion providing a field and the
synonym file would be helpful.

Thanks,
Roopa



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Multi words query time synonyms

[Dominique Bejean]

Hi,

I am trying multi words query time synonyms with Solr 6.6.2and
SynonymGraphFilterFactory filter as explain in this article
https://lucidworks.com/2017/04/18/multi-word-synonyms-solr-adds-query-time-support/

My field type is :



  
  
  
  
  
  


  
  
  
  
  
  
  

  


synonyms.txt contains the line

om, olympique de marseille


The order of words in my query has an impact on the generated query in
edismax

q={!edismax qf='name_text_gp' v=$qq}
&sow=false
&qq=...

with "qq=om maillot" or "qq=olympique de marseille maillot", I can see the
synonyms expansion. It is working as expected.

"parsedquery_toString":"+(((+name_text_gp:olympiqu +name_text_gp:marseil
+name_text_gp:maillot) name_text_gp:om))",
"parsedquery_toString":"+((name_text_gp:om (+name_text_gp:olympiqu
+name_text_gp:marseil +name_text_gp:maillot)))",


with "qq=maillot om" or "qq=maillot olympique de marseille", I can see the
same generated query

"parsedquery_toString":"+((name_text_gp:maillot) (name_text_gp:om))",
"parsedquery_toString":"+((name_text_gp:maillot) (name_text_gp:om))",

I don't understand these generated queries. The first one looks like the
synonym expansion is ignored, but the second one shows it is not ignored
and only the synonym term is used.


What is wrong in the way I am doing this ?

Regards

Dominique

-- 
Dominique Béjean
06 08 46 12 43

Re: Solr Swap space

[Susheel Kumar]

Thanks, Shawn, Eric.  I see that same using swapon -s.  Looks like during
the OS setup, it was set as 2 GB (Solr 6.0) and other 16GB (Solr 6.6)

Our 6.0 instance has been running since 1+ year but recently our monit
started reporting swap usage above 30% and Solr dashboard showing the
same.  I haven't been able to find what causes / why Solr is using the swap
space.  The index size is well within memory size to fit and we have double
the index size on our performance test machines but there is no usage of
swap space.

Thanks,
Susheel

On Wed, Feb 7, 2018 at 3:09 PM, Shawn Heisey  wrote:

> On 2/7/2018 12:01 PM, Susheel Kumar wrote:
>
>> Just trying to find where do we set swap space available to Solr process.
>> I
>> see in our 6.0 instances it was set to 2GB on and on 6.6 instances its set
>> to 16GB.
>>
>
> Solr has absolutely no involvement or control over swap space.  Neither
> does Java.  This is a function of your operating system's memory
> management, and is typically set up when you first install your OS.
>
> https://www.linux.com/news/all-about-linux-swap-space
> https://en.wikipedia.org/wiki/Paging#Windows_NT
>
> If your system is using swap space, it's a strong indication that you
> don't have enough memory installed.  If any of the memory that Solr uses is
> swapped out to disk, Solr performance is going to be REALLY bad.
>
> Thanks,
> Shawn
>

Re: Solr Swap space

[Susheel Kumar]

Sorry, I meant Emir.

On Fri, Feb 9, 2018 at 3:15 PM, Susheel Kumar  wrote:

> Thanks, Shawn, Eric.  I see that same using swapon -s.  Looks like during
> the OS setup, it was set as 2 GB (Solr 6.0) and other 16GB (Solr 6.6)
>
> Our 6.0 instance has been running since 1+ year but recently our monit
> started reporting swap usage above 30% and Solr dashboard showing the
> same.  I haven't been able to find what causes / why Solr is using the swap
> space.  The index size is well within memory size to fit and we have double
> the index size on our performance test machines but there is no usage of
> swap space.
>
> Thanks,
> Susheel
>
> On Wed, Feb 7, 2018 at 3:09 PM, Shawn Heisey  wrote:
>
>> On 2/7/2018 12:01 PM, Susheel Kumar wrote:
>>
>>> Just trying to find where do we set swap space available to Solr
>>> process. I
>>> see in our 6.0 instances it was set to 2GB on and on 6.6 instances its
>>> set
>>> to 16GB.
>>>
>>
>> Solr has absolutely no involvement or control over swap space.  Neither
>> does Java.  This is a function of your operating system's memory
>> management, and is typically set up when you first install your OS.
>>
>> https://www.linux.com/news/all-about-linux-swap-space
>> https://en.wikipedia.org/wiki/Paging#Windows_NT
>>
>> If your system is using swap space, it's a strong indication that you
>> don't have enough memory installed.  If any of the memory that Solr uses is
>> swapped out to disk, Solr performance is going to be REALLY bad.
>>
>> Thanks,
>> Shawn
>>
>
>

Re: Hard commits blocked | non-solrcloud v6.6.2

[mmb1234]

Shawn, Eric, 

Were you able to look at the thread dump ?
https://github.com/mohsinbeg/datadump/blob/master/threadDump-7pjql_1.zip

Or is there additional data I may provide.





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

RE: lat/long (location ) field context filters for autosuggestions

[Deepak Udapudi]

Hi all,

Any response on the below would be really helpful.

Thanks,
Deepak


-Original Message-
From: Deepak Udapudi [mailto:dudap...@delta.org] 
Sent: Thursday, February 08, 2018 2:02 PM
To: solr-user@lucene.apache.org
Subject: lat/long (location ) field context filters for autosuggestions

Hi all,

We are trying to apply context filter for autosuggestions by passing the 
lat/long of a particular location.
We know that, we can apply the context field for a particular field which takes 
one value.
How do we apply the same context filtering for a field which takes two values 
lat/long i.e location in our case.
Any pointers / suggestions to index/retrieve the suggestions would be really 
helpful.

Thanks in advance.

Regards,
Deepak




The information contained in this email message and any attachments is 
confidential and intended only for the addressee(s). If you are not an 
addressee, you may not copy or disclose the information, or act upon it, and 
you should delete it entirely from your email system. Please notify the sender 
that you received this email in error.


The information contained in this email message and any attachments is 
confidential and intended only for the addressee(s). If you are not an 
addressee, you may not copy or disclose the information, or act upon it, and 
you should delete it entirely from your email system. Please notify the sender 
that you received this email in error.

Re: solrcloud Auto-commit doesn't seem reliable

[Erick Erickson]

Do you by any chance have buffering turned on for CDCR? That parameter
is misleading. If true, tlogs will accumulate forever. The blanket
recommendation is becoming turn buffering off and leave it off, the
original intention there has been replaced really by bootstrapping.
Buffering was there for maintenance windows before bootstrapping was
put in place. However, even if this is an issue, it shouldn't be
affecting your commits.

This is puzzling. CDCR is just using tlogs as a queueing mechanism.
Docs are sent from the source to the target cluster, but once received
they're _supposed_ to be just like any other doc that's indexed, i.e.
when the first one is received it should trigger the start of the
autocommit intervals.

Is there any possibility that:
1> your config isn't correct? I'm guessing this is just a typo:
"autoSoftcommit", but worth checking. Besides, that's not your main
concern anyway.
2> your startup parameters override your solrconfig settings for hard
commit interval and set it to -1 or something? When you start Solr you
should see the result of all the various ways you can set these
intervals "rolled up", look for messages like:

INFO  org.apache.solr.update.CommitTracker; Hard AutoCommit: if
uncommited for 15000ms;
INFO  org.apache.solr.update.CommitTracker; Soft AutoCommit: disabled


There's some chance that the admin console is misleading, but you've
seen the behavior change when you commit so that's unlikely to be the
root of your issue either, but mentioning it in passing.

bq: We see relvancy scores get inconsistent when there are too many
deletes which seems to happen when hard commits don't happen

right, the hard commit will trigger merging, which in turn removes the
terms associated with deleted documents in the segments that are
merged which in turn changes your TF/IDF stats. So this is another
piece of evidence that your getting unexpected behavior. And your
tlogs will accumulate forever too.

This is the first time I've ever heard of this problem, so I'm still
thinking that this is something odd about your setup, but what it is
escapes me from what you've said so far.

I want to check one other thing: You say you've seen this  behavior in
4.10. CDCR wasn't introduced until considerably later, so what was the
scenario in the 4.10 case? Is my tangent for CDCR just a red herring?

Best,
Erick


On Fri, Feb 9, 2018 at 8:29 AM, Webster Homer  wrote:
> A little more background. Our production Solrclouds are populated via CDCR,
> CDCR does not replicate commits, Commits to the target clouds happen via
> autoCommit settings
>
> We see relvancy scores get inconsistent when there are too many deletes
> which seems to happen when hard commits don't happen.
>
> On Fri, Feb 9, 2018 at 10:25 AM, Webster Homer 
> wrote:
>
>> I we do have autoSoftcommit set to 3 seconds. It is NOT the visibility of
>> the records that is my primary concern. I am concerned about is the
>> accumulation of uncommitted tlog files and the larger number of deleted
>> documents.
>>
>> I am VERY familiar with the Solr documentation on this.
>>
>> On Fri, Feb 9, 2018 at 10:08 AM, Shawn Heisey  wrote:
>>
>>> On 2/9/2018 8:44 AM, Webster Homer wrote:
>>>
 I look at the latest timestamp on a record in the collection and see that
 it is over 24 hours old.

 I send a commit to the collection, and then see that the core is now
 current, and the segments are fewer. The commit worked

 This is the setting in solrconfig.xml
  ${solr.autoCommit.maxTime:6} <
 openSearcher>false 

>>>
>>> As recommended, you have openSearcher set to false.
>>>
>>> This means that these commits are NEVER going to make changes visible.
>>>
>>> Don't go and change openSearcher to true.  It is STRONGLY recommended to
>>> have openSearcher=false in your autoCommit settings.  The reason for this
>>> configuration is that it prevents the transaction log from growing out of
>>> control.  With openSearcher=false, those commits will be very fast.  This
>>> is because it's opening the searcher that's slow, not the process of
>>> writing data to disk.
>>>
>>> Here's the recommended reading on the subject:
>>>
>>> https://lucidworks.com/understanding-transaction-logs-softco
>>> mmit-and-commit-in-sorlcloud/
>>>
>>> For change visibility, configure autoSoftCommit, probably with a
>>> different interval than you have for autoCommit.  I would recommend a
>>> longer interval.  Or include the commitWithin parameter on at least some of
>>> your update requests.  Or send explicit commit requests, preferably as soft
>>> commits.
>>>
>>> Thanks,
>>> Shawn
>>>
>>
>>
>
> --
>
>
> This message and any attachment are confidential and may be privileged or
> otherwise protected from disclosure. If you are not the intended recipient,
> you must not copy this message or attachment or disclose the contents to
> any other person. If you have received this transmission in error, please
> notify the sender immediately and delete the mess

Re: Hard commits blocked | non-solrcloud v6.6.2

[Shawn Heisey]

On 2/8/2018 5:50 PM, mmb1234 wrote:

I then removed my custom  element from my solrconfig.xml and
both hard commit and /solr/admin/core hang issues seemed to go way for a
couple of hours.


The mergeScheduler config you had is not likely to be causing any 
issues.  It's a good config.  But removing it shouldn't do all that much 
for the things we've been discussing either.  It would be fine to put it 
back in, or leave it out.



Then I increased the ingestion load and the issue came back. Taking a heap
dump I now see lots of these and back to commit hanging. Full thread dump is
at https://github.com/mohsinbeg/datadump/blob/master/threadDump-7pjql_1.zip


There are over six thousand threads (which is a lot) that are in a 
WAITING state while trying to add documents to the index.  But I cannot 
tell what they are waiting for -- the hex ID values that the threads are 
waiting for are not mentioned as being held by any of the threads in the 
dump.


At the end of this message is an example of a stacktrace for one of 
those threads.  It doesn't appear to be the commits that are the problem 
here, but rather something keeping the indexing from happening ... 
except that you have said the index size is increasing, so I don't 
really know what's going on.


There are some other details I would like to see.

I would like to see a "top" display.  Run the top utility (not htop or 
any other variant), press shift-M to sort by memory, grab a screenshot 
of that display, and put it somewhere we can retrieve it.


Also, can you let Solr run for a while, preferably while recreating the 
problem, and then share the GC log that Solr creates?


Thanks,
Shawn


==

"qtp310656974-50242" #50242 prio=5 os_prio=0 tid=0x7e2120b35800 
nid=0xaf95 runnable [0x7f866155b000]

   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000268c4e590> (a 
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:871)
at 
org.apache.solr.update.DefaultSolrCoreState.lock(DefaultSolrCoreState.java:176)
at 
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:121)
at 
org.apache.solr.update.DirectUpdateHandler2.allowDuplicateUpdate(DirectUpdateHandler2.java:269)
at 
org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:242)
at 
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:194)
at 
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
at 
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at 
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1005)
at 
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:748)
at 
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at 
com.vmware.ops.data.solr.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:342)
at 
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at 
org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)
at 
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at 
com.vmware.ops.data.solr.processor.UnknownFieldNameMutatingUpdateProcessorFactory$1.processAdd(UnknownFieldNameMutatingUpdateProcessorFactory.java:91)
at 
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at 
com.vmware.ops.data.solr.processor.CopyTextFieldProcessorFactory$CopyTextFieldUpdateProcessor.processAdd(CopyTextFieldProcessorFactory.java:58)
at 
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at 
com.vmware.ops.data.solr.processor.TimestampExtractorUpdateProcessor.processAdd(UpdateTimestampProcessorFactory.java:67)
at 
com.vmware.ops.data.solr.handler.IngestionHandler.handleRequestBody(IngestionHandler.java:340)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
at 
com.vmware.ops.data.solr.handler.AggregateIngestionHandler.handle

Re: solrcloud Auto-commit doesn't seem reliable

[Shawn Heisey]

On 2/9/2018 9:29 AM, Webster Homer wrote:

A little more background. Our production Solrclouds are populated via CDCR,
CDCR does not replicate commits, Commits to the target clouds happen via
autoCommit settings

We see relvancy scores get inconsistent when there are too many deletes
which seems to happen when hard commits don't happen.

On Fri, Feb 9, 2018 at 10:25 AM, Webster Homer 
wrote:


I we do have autoSoftcommit set to 3 seconds. It is NOT the visibility of
the records that is my primary concern. I am concerned about is the
accumulation of uncommitted tlog files and the larger number of deleted
documents.


For the deleted documents:  Have you ever done an optimize on the 
collection?  If so, you're going to need to re-do the optimize regularly 
to keep deleted documents from growing out of control.  See this issue 
for a very technical discussion about it:


https://issues.apache.org/jira/browse/LUCENE-7976

Deleted documents probably aren't really related to what we've been 
discussing.  That shouldn't really be strongly affected by commit settings.


-

A 3 second autoSoftCommit is VERY aggressive.   If your soft commits are 
taking longer than 3 seconds to complete, which is often what happens, 
then that will lead to problems.  I wouldn't expect it to cause the 
kinds of problems you describe, though.  It would manifest as Solr 
working too hard, logging warnings or errors, and changes taking too 
long to show up.


Assuming that the config for autoSoftCommit doesn't have the typo that 
Erick mentioned.




I have never used CDCR, so I know very little about it.  But I have seen 
reports on this mailing list saying that transaction logs never get 
deleted when CDCR is configured.


Below is a link to a mailing list discussion related to CDCR not 
deleting transaction logs.  Looks like for it to work right a buffer 
needs to be disabled, and there may also be problems caused by not 
having a complete zkHost string in the CDCR config:


http://lucene.472066.n3.nabble.com/CDCR-how-to-deal-with-the-transaction-log-files-td4345062.html

Erick also mentioned this.

Thanks,
Shawn

Re: High CPU and Physical Memory Usage in solr with 4000 user load

[Shawn Heisey]

On 2/9/2018 12:48 AM, rubi.hali wrote:

As threads are blocked , our cpu is reaching to 90% even even when heap or
os memory is not getting used at all.

One of the BLOCKED Thread Snippet
Most of the threads are blocked either on Jetty Connector or FieldCacheIMPL
for DOC values.  Slave1ThreadDump5.txt

Slave1ThreadDump4.txt



Threads being blocked shouldn't cause high CPU.  It would generally be 
caused by whatever is causing the threads to be blocked.


I uploaded the Slave1ThreadDump5.txt file from the first link to the 
http://fastthread.io website.


The analysis noticed that 45 threads were blocked by another thread.  At 
the end of this message is the stacktrace for the thread blocking the 
others.


What jumps out at me is the mention of a custom plugin for Solr in the 
stacktrace.  Here's the method mentioned:


com.ril.solr.request.handler.RILMultipleQuerySearchHandler.handleRequestBody

Could there be a bug in that handler causing problems?  I can't say for 
sure, but this IS something out of the ordinary.


This thread is blocking other threads, but it is itself blocked.   Says 
it's waiting for 0x7f24d881e000, but this entry does not appear 
anywhere else in the thread dump.  So I can't tell what's actually 
blocking it.


I also tried putting the thread dump into this page:

https://spotify.github.io/threaddump-analyzer/

The analysis there says that qtp1205044462-196 is "inconsistent" and the 
mouseover for that says that the Thread is blocked on "BLOCKED (on 
object monitor)" without waiting for anything.


Can't find anything definitive, but it MIGHT be blocked by java's own 
internal operations.  Things like memory allocations, GC pauses, etc.


I noticed that the stacktrace includes mention of Solr's facet classes, 
suggesting that these queries include facets or grouping.  Facets and 
grouping can require very large amounts of heap, especially if the 
fields being used do not have docValues enabled.  You may need to make 
your heap larger so that there are fewer garbage collection events.


I have no idea what the custom search handler is going to do to Solr's 
memory requirements.


Solr should create GC logs if you have started it with the included 
script.  Can you share a GC log that includes the time the load testing 
is underway and the problem appears?


Can you log onto the Linux server, run the "top" program (not htop or 
any other variant), press shift-M to sort the list by memory, grab a 
screenshot, and share it?


Thanks,
Shawn



"qtp1205044462-196" #196 prio=5 os_prio=0 tid=0x7f2374002800 
nid=0x5171 waiting for monitor entry [0x7f24d881e000]

   java.lang.Thread.State: BLOCKED (on object monitor)
	at 
org.apache.solr.uninverting.FieldCacheImpl$Cache.get(FieldCacheImpl.java:167)

- locked <0x000642cd7568> (a java.util.WeakHashMap)
	at 
org.apache.solr.uninverting.FieldCacheImpl.getTermsIndex(FieldCacheImpl.java:767)
	at 
org.apache.solr.uninverting.FieldCacheImpl.getTermsIndex(FieldCacheImpl.java:747)
	at 
org.apache.solr.uninverting.UninvertingReader.getSortedDocValues(UninvertingReader.java:319)
	at 
org.apache.lucene.index.FilterLeafReader.getSortedDocValues(FilterLeafReader.java:448)

at org.apache.lucene.index.DocValues.getSorted(DocValues.java:262)
	at 
org.apache.lucene.search.grouping.term.TermGroupFacetCollector$SV.doSetNextReader(TermGroupFacetCollector.java:128)
	at 
org.apache.lucene.search.SimpleCollector.getLeafCollector(SimpleCollector.java:33)

at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:659)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:472)
	at 
org.apache.solr.request.SimpleFacets.getGroupedCounts(SimpleFacets.java:692)
	at 
org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:476)
	at 
org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:405)
	at 
org.apache.solr.request.SimpleFacets.lambda$getFacetFieldCounts$0(SimpleFacets.java:803)
	at 
org.apache.solr.request.SimpleFacets$$Lambda$207/1284277177.call(Unknown 
Source)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.request.SimpleFacets$3.execute(SimpleFacets.java:742)
	at 
org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:818)
	at 
org.apache.solr.handler.component.FacetComponent.getFacetCounts(FacetComponent.java:330)
	at 
org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:274)
	at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:296)
	at 
com.ril.solr.request.handler.RILMultipleQuerySearchHandler.handleRequestBody(RILMultipleQuerySearchHandler.java:39)
	at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)

at org.apache.solr.core.Sol

Re: Solr Swap space

[Emir Arnautović]

No worries, I don't mind being confused with Erick ;)

Emir

On Feb 9, 2018 9:16 PM, "Susheel Kumar"  wrote:

> Sorry, I meant Emir.
>
> On Fri, Feb 9, 2018 at 3:15 PM, Susheel Kumar 
> wrote:
>
> > Thanks, Shawn, Eric.  I see that same using swapon -s.  Looks like during
> > the OS setup, it was set as 2 GB (Solr 6.0) and other 16GB (Solr 6.6)
> >
> > Our 6.0 instance has been running since 1+ year but recently our monit
> > started reporting swap usage above 30% and Solr dashboard showing the
> > same.  I haven't been able to find what causes / why Solr is using the
> swap
> > space.  The index size is well within memory size to fit and we have
> double
> > the index size on our performance test machines but there is no usage of
> > swap space.
> >
> > Thanks,
> > Susheel
> >
> > On Wed, Feb 7, 2018 at 3:09 PM, Shawn Heisey 
> wrote:
> >
> >> On 2/7/2018 12:01 PM, Susheel Kumar wrote:
> >>
> >>> Just trying to find where do we set swap space available to Solr
> >>> process. I
> >>> see in our 6.0 instances it was set to 2GB on and on 6.6 instances its
> >>> set
> >>> to 16GB.
> >>>
> >>
> >> Solr has absolutely no involvement or control over swap space.  Neither
> >> does Java.  This is a function of your operating system's memory
> >> management, and is typically set up when you first install your OS.
> >>
> >> https://www.linux.com/news/all-about-linux-swap-space
> >> https://en.wikipedia.org/wiki/Paging#Windows_NT
> >>
> >> If your system is using swap space, it's a strong indication that you
> >> don't have enough memory installed.  If any of the memory that Solr
> uses is
> >> swapped out to disk, Solr performance is going to be REALLY bad.
> >>
> >> Thanks,
> >> Shawn
> >>
> >
> >
>

Re: Hard commits blocked | non-solrcloud v6.6.2

[mmb1234]

Ran /solr/58f449cec94a2c75-core-248/admin/luke at 7:05pm PST

It showed "lastModified: 2018-02-10T02:25:08.231Z" indicating commit blocked
for about 41 mins.
Hard commit is set as 10secs in solrconfig.xml

Other cores are also now blocked.

https://jstack.review analysis of the thread dump says "Potential Deadlock"
for three 3 cores.

0x00027afca8e0 
ConcurrentMergeScheduler 
Possible deadlock
Held by commitScheduler-26-thread-1 awaiting notification , holding [
0x00027afca8e0 ]

0x00027fd19080 
ConcurrentMergeScheduler 
Possible deadlock   
Held by commitScheduler-19-thread-1 awaiting notification , holding [
0x00027fd19080 0x0002615ada20 ]

0x000280da09c8 
ConcurrentMergeScheduler 
Possible deadlock   
Held by commitScheduler-24-thread-1 awaiting notification , holding [
0x000280da09c8 0x00028e0cbf10 ]

Thread dump and top output are from that condition are available at
https://github.com/mohsinbeg/datadump/tree/master/solr58f449cec94a2c75_core_248/solr58f449cec94a2c75_core_248


848 thread(s) have this kind of stack:
---
at java.lang.Object.wait(Native Method)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doStall(ConcurrentMergeScheduler.java:616)
at
org.apache.lucene.index.ConcurrentMergeScheduler.maybeStall(ConcurrentMergeScheduler.java:602)
...
 at
org.apache.solr.update.DirectUpdateHandler2.allowDuplicateUpdate(DirectUpdateHandler2.java:276)
...



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Are flushed (but not committed yet) segments mutable?

[Nawab Zada Asad Iqbal]

Hi,

When a segment is flushed to disk because it is exceeding available memory,
is it sill updated when new documents are added? I also read somewhere that
a segment is not committed even if it is flushed. How is a
flushed-but-not-committed segment different from a committed segment?

For example, my hard-commit is scheduled for every 30 seconds, but many
segments are flushed during this interval. Are they flushed as in-memory
data structures (which will keep them optimal for updates) or are they
immutable?

I also see some segment merges before the hard-commit executes, which make
me think that flush converts the in-memory data-structures into Lucene
segment.

Thanks
Nawab

Unable to get the results in ordered manner from SOLR

[Anupam Bhattacharya]

For me the SOLR sort was working fine with boost increments of 1. But after
some recent changes to do atomic updates to existing SOLR document now when
I try to get ordered results I get it in improper order. I have tried to
keep large difference in the boosting values between individual terms but
its very inconsistent.

Below is the query with debugQuery=on.

{
  "responseHeader":{
"status":0,
"QTime":19,
"params":{
  "q":"pageID:(\"254eead5-0c17-11e8-a4e0-067beceeb88f\"^14501 OR
\"f05a7d41-0b19-11e8-8f16-067beceeb88f\"^14001 OR
\"5d368d4f-0c16-11e8-a4e0-067beceeb88f\"^13501 OR
\"ebcdc438-0702-11e8-9ea1-068a0dc6bc8f\"^13001 OR
\"f\r\n72e001a-0c1a-11e8-a4e0-067beceeb88f\"^12501 OR
\"dd73b9b6-0c17-11e8-a4e0-067beceeb88f\"^12001 OR
\"bbd40922-0c8c-11e8-9b48-067beceeb88f\"^11501 OR
\"d00791e9-0c1a-11e8-a4e0-067beceeb88f\"^11001 OR
\"fbe95d\r\nb9-0bf8-11e8-a5c6-067beceeb88f\"^10501 OR
\"8778980f-0d8c-11e8-b7a8-067beceeb88f\"^10001 OR
\"e48e4073-02a4-11e8-a9a6-068a0dc6bc8f\"^9501 OR
\"3b91f3ab-00ed-11e8-ac1d-067beceeb88f\"^9001 OR
\"e878f0a7-0d98\r\n-11e8-916e-067beceeb88f\"^8501 OR
\"beca0178-0bf6-11e8-a5c6-067beceeb88f\"^8001 OR
\"864a7dee-01e4-11e8-80b0-067beceeb88f\"^7501 OR
\"70b34ec1-ff38-11e7-b9c9-067beceeb88f\"^7001 OR
\"ddd51847-0d8d-11e8-b7a\r\n8-067beceeb88f\"^6501 OR
\"ad8d954d-000a-11e8-92a3-067beceeb88f\"^6001 OR
\"93f24713-0ae4-11e8-866d-068a0dc6bc8f\"^5501 OR
\"87033dd0-ff37-11e7-b9c9-067beceeb88f\"^5001 OR
\"65b079c8-0c19-11e8-a4e0-067bece\r\neb88f\"^4501 OR
\"9c5f0007-0c18-11e8-a4e0-067beceeb88f\"^4001 OR
\"0a5796d5-0d7f-11e8-bf64-067beceeb88f\"^3501 OR
\"b3800b06-0104-11e8-99f2-067beceeb88f\"^3001 OR
\"9ed6136c-058c-11e8-8951-067beceeb88f\"^25\r\n01 OR
\"af34b7a5-0102-11e8-99f2-067beceeb88f\"^2001 OR
\"9f30cfa1-fe59-11e7-adca-067beceeb88f\"^1501 OR
\"91a40514-ff3e-11e7-b9c9-067beceeb88f\"^1001 OR
\"1aed96f2-0b44-11e8-94a2-067beceeb88f\"^501 OR
\"51d\r\n42489-0049-11e8-8628-067beceeb88f\"^1)\r\n",
  "sort=score desc":"",
  "fl":"pageID,score",
  "rows":"100",
  "debugQuery":"on"}},
  "response":{"numFound":24,"start":0,"maxScore":79148.87,"docs":[
  {
"pageID":"254eead5-0c17-11e8-a4e0-067beceeb88f",
"score":79148.87},
  {
"pageID":"5d368d4f-0c16-11e8-a4e0-067beceeb88f",
"score":73690.7},
  {
"pageID":"f05a7d41-0b19-11e8-8f16-067beceeb88f",
"score":69267.71},
  {
"pageID":"dd73b9b6-0c17-11e8-a4e0-067beceeb88f",
"score":65503.45},
  {
"pageID":"ebcdc438-0702-11e8-9ea1-068a0dc6bc8f",
"score":64320.375},
  {
"pageID":"bbd40922-0c8c-11e8-9b48-067beceeb88f",
"score":62774.367},
  {
"pageID":"d00791e9-0c1a-11e8-a4e0-067beceeb88f",
"score":60045.28},
  {
"pageID":"8778980f-0d8c-11e8-b7a8-067beceeb88f",
"score":54587.12},
  {
"pageID":"e48e4073-02a4-11e8-a9a6-068a0dc6bc8f",
"score":47004.684},
  {
"pageID":"3b91f3ab-00ed-11e8-ac1d-067beceeb88f",
"score":44531.01},
  {
"pageID":"beca0178-0bf6-11e8-a5c6-067beceeb88f",
"score":43670.79},
  {
"pageID":"864a7dee-01e4-11e8-80b0-067beceeb88f",
"score":37110.0},
  {
"pageID":"70b34ec1-ff38-11e7-b9c9-067beceeb88f",
"score":34636.332},
  {
"pageID":"ad8d954d-000a-11e8-92a3-067beceeb88f",
"score":32754.455},
  {
"pageID":"87033dd0-ff37-11e7-b9c9-067beceeb88f",
"score":27296.29},
  {
"pageID":"93f24713-0ae4-11e8-866d-068a0dc6bc8f",
"score":27215.32},
  {
"pageID":"9c5f0007-0c18-11e8-a4e0-067beceeb88f",
"score":21838.123},
  {
"pageID":"0a5796d5-0d7f-11e8-bf64-067beceeb88f",
"score":19109.04},
  {
"pageID":"b3800b06-0104-11e8-99f2-067beceeb88f",
"score":14846.969},
  {
"pageID":"af34b7a5-0102-11e8-99f2-067beceeb88f",
"score":10921.79},
  {
"pageID":"9f30cfa1-fe59-11e7-adca-067beceeb88f",
"score":7425.958},
  {
"pageID":"91a40514-ff3e-11e7-b9c9-067beceeb88f",
"score":5463.6245},
  {
"pageID":"1aed96f2-0b44-11e8-94a2-067beceeb88f",
"score":2734.5413},
  {
"pageID":"9ed6136c-058c-11e8-8951-067beceeb88f",
"score":136.45415}]
  },
  "debug":{

"rawquerystring":"pageID:(\"254eead5-0c17-11e8-a4e0-067beceeb88f\"^14501 OR
\"f05a7d41-0b19-11e8-8f16-067beceeb88f\"^14001 OR
\"5d368d4f-0c16-11e8-a4e0-067beceeb88f\"^13501 OR
\"ebcdc438-0702-11e8-9ea1-068a0dc6bc8f\"^13001 OR
\"f\r\n72e001a-0c1a-11e8-a4e0-067beceeb88f\"^12501 OR
\"dd73b9b6-0c17-11e8-a4e0-067beceeb88f\"^12001 OR
\"bbd40922-0c8c-11e8-9b48-067beceeb88f\"^11501 OR
\"d00791e9-0c1a-11e8-a4e0-067beceeb88f\"^11001 OR
\"fbe95d\r\nb9-0bf8-11e8-a5c6-067beceeb88f\"^10501 OR
\"8778980f-0d8c-11e8-b7a8-067beceeb88f\"^10001 OR
\"e48e4073-02a4-11e8-a9a6-068a0dc6bc8f\"^9501 OR
\"3b91f3ab-00ed-11e8-ac1d-067beceeb88f\"^9001 OR
\"e8

Collection API Snapshots and Restore

[Nicolas Bélisle]

It is possible to restore a snapshot without exporting it using the
Collection API ?

We have a very large index and want to provide a quick option for restore
(instead of exporting + restoring + fixing the topology).

I think about developing a script that would read the snapshot's XML
description and manually delete files not part of that snapshot on each
core.

Is there another solution ?

Nicolas

Re: Unable to get the results in ordered manner from SOLR

[Anupam Bhattacharya]

I further tried to understand the results I see since the idf value is
smaller higher boost is not helping. Can we ensure that idf value should
not impact the ordering.


73690.7 = sum of:
  73690.7 = weight(pageID:5d368d4f-0c16-11e8-a4e0-067beceeb88f in 36)
[SchemaSimilarity], result of:
73690.7 = score(doc=36,freq=1.0 = termFreq=1.0
), product of:
  13501.0 = boost
  5.458166 = idf, computed as log(1 + (docCount - docFreq + 0.5) /
(docFreq + 0.5)) from:
1.0 = docFreq
351.0 = docCount
  1.0 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1) from:
1.0 = termFreq=1.0
1.2 = parameter k1
0.0 = parameter b (norms omitted for field)


69267.71 = sum of:
  69267.71 = weight(pageID:f05a7d41-0b19-11e8-8f16-067beceeb88f in 2)
[SchemaSimilarity], result of:
69267.71 = score(doc=2,freq=1.0 = termFreq=1.0
), product of:
  14001.0 = boost
  4.9473405 = idf, computed as log(1 + (docCount - docFreq + 0.5) /
(docFreq + 0.5)) from:
2.0 = docFreq
351.0 = docCount
  1.0 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1) from:
1.0 = termFreq=1.0
1.2 = parameter k1
0.0 = parameter b (norms omitted for field)



On Sat, Feb 10, 2018 at 10:58 AM, Anupam Bhattacharya 
wrote:

> For me the SOLR sort was working fine with boost increments of 1. But
> after some recent changes to do atomic updates to existing SOLR document
> now when I try to get ordered results I get it in improper order. I have
> tried to keep large difference in the boosting values between individual
> terms but its very inconsistent.
>
> Below is the query with debugQuery=on.
>
> {
>   "responseHeader":{
> "status":0,
> "QTime":19,
> "params":{
>   "q":"pageID:(\"254eead5-0c17-11e8-a4e0-067beceeb88f\"^14501 OR
> \"f05a7d41-0b19-11e8-8f16-067beceeb88f\"^14001 OR
> \"5d368d4f-0c16-11e8-a4e0-067beceeb88f\"^13501 OR
> \"ebcdc438-0702-11e8-9ea1-068a0dc6bc8f\"^13001 OR
> \"f\r\n72e001a-0c1a-11e8-a4e0-067beceeb88f\"^12501 OR
> \"dd73b9b6-0c17-11e8-a4e0-067beceeb88f\"^12001 OR
> \"bbd40922-0c8c-11e8-9b48-067beceeb88f\"^11501 OR
> \"d00791e9-0c1a-11e8-a4e0-067beceeb88f\"^11001 OR
> \"fbe95d\r\nb9-0bf8-11e8-a5c6-067beceeb88f\"^10501 OR
> \"8778980f-0d8c-11e8-b7a8-067beceeb88f\"^10001 OR
> \"e48e4073-02a4-11e8-a9a6-068a0dc6bc8f\"^9501 OR
> \"3b91f3ab-00ed-11e8-ac1d-067beceeb88f\"^9001 OR
> \"e878f0a7-0d98\r\n-11e8-916e-067beceeb88f\"^8501 OR
> \"beca0178-0bf6-11e8-a5c6-067beceeb88f\"^8001 OR
> \"864a7dee-01e4-11e8-80b0-067beceeb88f\"^7501 OR
> \"70b34ec1-ff38-11e7-b9c9-067beceeb88f\"^7001 OR
> \"ddd51847-0d8d-11e8-b7a\r\n8-067beceeb88f\"^6501 OR
> \"ad8d954d-000a-11e8-92a3-067beceeb88f\"^6001 OR
> \"93f24713-0ae4-11e8-866d-068a0dc6bc8f\"^5501 OR
> \"87033dd0-ff37-11e7-b9c9-067beceeb88f\"^5001 OR
> \"65b079c8-0c19-11e8-a4e0-067bece\r\neb88f\"^4501 OR
> \"9c5f0007-0c18-11e8-a4e0-067beceeb88f\"^4001 OR
> \"0a5796d5-0d7f-11e8-bf64-067beceeb88f\"^3501 OR
> \"b3800b06-0104-11e8-99f2-067beceeb88f\"^3001 OR
> \"9ed6136c-058c-11e8-8951-067beceeb88f\"^25\r\n01 OR
> \"af34b7a5-0102-11e8-99f2-067beceeb88f\"^2001 OR
> \"9f30cfa1-fe59-11e7-adca-067beceeb88f\"^1501 OR
> \"91a40514-ff3e-11e7-b9c9-067beceeb88f\"^1001 OR
> \"1aed96f2-0b44-11e8-94a2-067beceeb88f\"^501 OR
> \"51d\r\n42489-0049-11e8-8628-067beceeb88f\"^1)\r\n",
>   "sort=score desc":"",
>   "fl":"pageID,score",
>   "rows":"100",
>   "debugQuery":"on"}},
>   "response":{"numFound":24,"start":0,"maxScore":79148.87,"docs":[
>   {
> "pageID":"254eead5-0c17-11e8-a4e0-067beceeb88f",
> "score":79148.87},
>   {
> "pageID":"5d368d4f-0c16-11e8-a4e0-067beceeb88f",
> "score":73690.7},
>   {
> "pageID":"f05a7d41-0b19-11e8-8f16-067beceeb88f",
> "score":69267.71},
>   {
> "pageID":"dd73b9b6-0c17-11e8-a4e0-067beceeb88f",
> "score":65503.45},
>   {
> "pageID":"ebcdc438-0702-11e8-9ea1-068a0dc6bc8f",
> "score":64320.375},
>   {
> "pageID":"bbd40922-0c8c-11e8-9b48-067beceeb88f",
> "score":62774.367},
>   {
> "pageID":"d00791e9-0c1a-11e8-a4e0-067beceeb88f",
> "score":60045.28},
>   {
> "pageID":"8778980f-0d8c-11e8-b7a8-067beceeb88f",
> "score":54587.12},
>   {
> "pageID":"e48e4073-02a4-11e8-a9a6-068a0dc6bc8f",
> "score":47004.684},
>   {
> "pageID":"3b91f3ab-00ed-11e8-ac1d-067beceeb88f",
> "score":44531.01},
>   {
> "pageID":"beca0178-0bf6-11e8-a5c6-067beceeb88f",
> "score":43670.79},
>   {
> "pageID":"864a7dee-01e4-11e8-80b0-067beceeb88f",
> "score":37110.0},
>   {
> "pageID":"70b34ec1-ff38-11e7-b9c9-067beceeb88f",
> "score":34636.332},
>   {
> "pageID":"ad8d954d-000a-11e8-92a3-067beceeb88f",
> "score":32754.455},
>   {
> "pageID":"87033dd0-ff37-11e7-b9c9-067beceeb88f",
> "score":27296.29},
> 

Re: ./fs-manager process run under solr

[Randall Chamberlin]

I am experiencing this too.  For me the "solr" user is running "fs-manager"
from with the directory "/var/tmp/.X1M-Unix.  There is a "config.json",
"out.log" and "xmrig.log" file present.  The json looks like this:
{
"algo": "cryptonight",
"av": 0,
"background": true,
"colors": false,
"cpu-affinity": null,
"cpu-priority": null,
"donate-level": 2,
"log-file": "xmrig.log",
"max-cpu-usage": 85,
"print-time": 60,
"retries": 2,
"retry-pause": 3,
"safe": false,
"syslog": false,
"threads": null,
"pools": [
{
"url": "pool-proxy.com:8080",
"user": "user",
"pass": "x",
"keepalive": true,
"nicehash": false
}
]
}



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html