Solr Document Update issues

[aparana bhatt]

Hi ,

I have been facing lot of issues in using solr update functionality .
Multitude of requests respond with










* org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
Error from server at http://192.169.33.86/solr/cms
: Expected mime type
application/octet-stream but got text/html. 502 Proxy
ErrorProxy ErrorThe proxy server received
an invalid^Mresponse from an upstream server.^MThe proxy server could
not handle the request POST /solr/cms/update.Reason:
Error reading from remote server*

Used solr version -> 6.5.0  Type -> master/Slave config
Error in solr.log ->



























*2020-10-07 05:43:50.639 WARN  (qtp142261320-27831) [   x:cms]
o.a.s.c.SolrCore slow: [cms]  webapp=/solr path=/update
params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}
status=0 QTime=443272020-10-07 05:43:50.640 WARN  (qtp142261320-27837) [
x:cms] o.a.s.u.DefaultSolrCoreState WARNING - Dangerous
interruptjava.lang.InterruptedExceptionat
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326)
  at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:871)
  at
org.apache.solr.update.DefaultSolrCoreState.lock(DefaultSolrCoreState.java:167)
  at
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:112)
  at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:618)
  at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:93)
  at
org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)
  at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1895)
  at
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1872)
  at
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:68)
  at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:72)
  at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
  at org.apache.solr.core.SolrCore.execute(SolrCore.java:2440)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
  at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
  at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
  at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
  at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
  at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
  at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
  at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
  at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
  at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)*



The rate of update query on master solr is 6 request per min only .
Solr also slows down and search becomes really slow .
I don't understand where to look for an issue .
I have tried to check various parameters in update request , if I do
softcommit=true and commit =false then updates do not reflect , so i have
set below options ->

UpdateRequest updateRequest = new UpdateRequest();
  updateRequest.setAction( UpdateRequest.ACTION.COMMIT, true,
true);
waitsearcher=true ,
waitflush=true .

I do not get what is causing the issue . Kindly suggest .
Also I could not find much help from internet about given issues as well .


-- 
Regards

Aparana Bhatt

Solr 8.6.2 Facets query for Nested documents

[Abhay Kumar]

Hello Team,

I have sync following nested document in Solr 8.6.2.

{
   "id":"NCT04372953",
   "title":"Positive End-Expiratory Pressure (PEEP) Levels During Resuscitation 
of Preterm Infants at Birth (The POLAR Trial) ",
   "phase":"N/A",
   "status":"Not yet recruiting",
   "studytype":"Interventional",
   "SponsorName":[
  "Murdoch Childrens Research Institute|Children''s Hospital of 
Philadelphia|University of Amsterdam"
   ],
   "SponsorRole":[
  "lead|collaborator"
   ],
   "source":"Murdoch Childrens Research Institute",
   "sponsorrole":[
  "lead",
  "collaborator"
   ],
   "sponsorname":[
  "Murdoch Childrens Research Institute",
  "Children''s Hospital of Philadelphia",
  "University of Amsterdam"
   ],
   "investigatorsaffiliation":"",
   "investigatorname":[
  ""
   ],
   "therapeuticareaname":"",
   "text_suggest":[
  ""
   ],
   "investigatorrole":"",
   "_version_":1680437253090836480,
   "sites":{
  "id":"51002566",
  "facilitytype":"Hospital",
  "facilityname":"The Royal Women''s Hospital, Melbourne Australia",
  "facilitycountry":"Australia",
  "facilitystate":"Victoria",
  "facilitycity":"Parkville",
  "nodetype":"cnode",
  "facilityzip":"",
  "_nest_parent_":"NCT04372953",
  "phase":"",
  "studytype":"",
  "investigatorsaffiliation":"",
  "source":"",
  "title":"",
  "sponsorrole":[
 ""
  ],
  "investigatorname":[
 ""
  ],
  "therapeuticareaname":"",
  "text_suggest":[
 ""
  ],
  "investigatorrole":"",
  "sponsorname":[
 ""
  ],
  "status":"",
  "_version_":1680437253090836480
   },
   "investigators":[
  {
 "id":"6300662",
 "investigatorname":[
"Louise Owen"
 ],
 "nodetype":"cnode",
 "investigatorrole":"Principal Investigator",
 "investigatorsaffiliation":"The Royal Women''s Hospital, Melbourne 
Australia",
 "CongressScore":"",
 "TrialsScore":"Low",
 "PublicationScore":"",
 "_nest_parent_":"NCT04372953",
 "phase":"",
 "studytype":"",
 "source":"",
 "title":"",
 "sponsorrole":[
""
 ],
 "therapeuticareaname":"",
 "text_suggest":[
""
 ],
 "sponsorname":[
""
 ],
 "status":"",
 "_version_":1680437253090836480
  },
  {
 "id":"6426782",
 "investigatorname":[
"David Tingay, MBBS FRACP"
 ],
 "nodetype":"cnode",
 "investigatorrole":"Study Chair",
 "investigatorsaffiliation":"Royal Children''s Hospital, Melbourne 
Australia",
 "CongressScore":"",
 "TrialsScore":"",
 "PublicationScore":"",
 "_nest_parent_":"NCT04372953",
 "phase":"",
 "studytype":"",
 "source":"",
 "title":"",
 "sponsorrole":[
""
 ],
 "therapeuticareaname":"",
 "text_suggest":[
""
 ],
 "sponsorname":[
""
 ],
 "status":"",
 "_version_":1680437253090836480
  },
  {
 "id":"7663364",
 "investigatorname":[
"Omar Kamlin"
 ],
 "nodetype":"cnode",
 "investigatorrole":"Principal Investigator",
 "investigatorsaffiliation":"The Royal Women''s Hospital, Melbourne 
Australia",
 "CongressScore":"",
 "TrialsScore":"Low",
 "PublicationScore":"",
 "_nest_parent_":"NCT04372953",
 "phase":"",
 "studytype":"",
 "source":"",
 "title":"",
 "sponsorrole":[
""
 ],
 "therapeuticareaname":"",
 "text_suggest":[
""
 ],
 "sponsorname":[
""
 ],
 "status":"",
 "_version_":1680437253090836480
  }
   ],
   "therapeuticareas":[
  {
 "id":"ta-0-NCT04372953",
 "therapeuticareaname":"Premature Birth",
 "text_prefixauto":"Premature Birth",
 "text_suggest":[
"Premature Birth"
 ],
 "diseaseareas":[
""
 ],
 "nodetype":"cnode",
 "_nest_parent_":"NCT04372953",
 "phase":"",
 "studytype":"",
 "investigatorsaffiliation":"",
 "source":"",
 "title":"",
 "sponsorrole":[
""
 ],
 "investigatorname":[
""
 ],
 "investigatorrole":"",
 "sponsorname":[
""
 ],
 "status":"",
 "_version_":1680437253090836480,
 "therapeuticareaname_facet":"Premature Birth",
 "diseaseareas_facet":[
""
 ]
  },
  {
 "id":"ta-1-NCT04372953",
 "therapeuticareaname":"Lung Injury",
 "text_prefixauto":"Lung Injury",
 "tex

Re: Memory line in status output

[Ryan W]

This is how I start solr:

/opt/solr/bin/solr start

In my /etc/default/solr.in.sh, I have this...

GC_TUNE=" \
-XX:+UseG1GC \
-XX:+ParallelRefProcEnabled \
-XX:G1HeapRegionSize=8m \
-XX:MaxGCPauseMillis=200 \
-XX:+UseLargePages \
-XX:+AggressiveOpts \
"

But I don't know how to tell if Solr is using that file.

In my /opt/solr/bin there is no solr.in.sh, but there is a solr.in.sh.orig
-- perhaps I should copy my /etc/default/solr.in.sh to /opt/solr/bin ?

I am running Linux (RHEL).  The Solr version is 7.7.2.  Solr 8.x is not
compatible with my application.

Thank you.


On Mon, Oct 12, 2020 at 9:46 PM Shawn Heisey  wrote:

> On 10/12/2020 5:11 PM, Ryan W wrote:
> > Thanks.  How do I activate the G1GC collector?  Do I do this by editing a
> > config file, or by adding a parameter when I start solr?
> >
> > Oracle's docs are pointing me to a file that supposedly is at
> > instance-dir/OUD/config/java.properties, but I don't have that path.  I
> am
> > not sure what is meant by instance-dir here, but perhaps it means my JRE
> > install, which is at
> > /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre -- but
> > there is no "OUD" directory in this location.
>
> The collector is chosen by the startup options given to Java, in this
> case by the start script for Solr.  I've never heard of it being set by
> a config in the JRE.
>
> In Solr 7, the start script defaults to the CMS collector.  We have
> updated that to G1 in the latest Solr 8.x versions, because CMS has been
> deprecated by Oracle.
>
> Adding the following lines to the correct solr.in.sh would change the
> garbage collector to G1.  I got this from the "bin/solr" script in Solr
> 8.5.1:
>
>GC_TUNE=('-XX:+UseG1GC' \
>  '-XX:+PerfDisableSharedMem' \
>  '-XX:+ParallelRefProcEnabled' \
>  '-XX:MaxGCPauseMillis=250' \
>  '-XX:+UseLargePages' \
>  '-XX:+AlwaysPreTouch')
>
> If you used the service installer script to install Solr, then the
> correct file to add this to is usually /etc/default/solr.in.sh ... but
> if you did the install manually, it may be in the same bin directory
> that contains the solr script itself.  Your initial message says the
> solr home is /opt/solr/server/solr so I am assuming it's not running on
> Windows.
>
> Thanks,
> Shawn
>

Re: Memory line in status output

[Ryan W]

Or, perhaps if I start solr like so

service solr start

...it will use the solr.in.sh at /etc/default/solr.in.sh ?



On Tue, Oct 13, 2020 at 11:19 AM Ryan W  wrote:

> This is how I start solr:
>
> /opt/solr/bin/solr start
>
> In my /etc/default/solr.in.sh, I have this...
>
> GC_TUNE=" \
> -XX:+UseG1GC \
> -XX:+ParallelRefProcEnabled \
> -XX:G1HeapRegionSize=8m \
> -XX:MaxGCPauseMillis=200 \
> -XX:+UseLargePages \
> -XX:+AggressiveOpts \
> "
>
> But I don't know how to tell if Solr is using that file.
>
> In my /opt/solr/bin there is no solr.in.sh, but there is a
> solr.in.sh.orig -- perhaps I should copy my /etc/default/solr.in.sh to
> /opt/solr/bin ?
>
> I am running Linux (RHEL).  The Solr version is 7.7.2.  Solr 8.x is not
> compatible with my application.
>
> Thank you.
>
>
> On Mon, Oct 12, 2020 at 9:46 PM Shawn Heisey 
> wrote:
>
>> On 10/12/2020 5:11 PM, Ryan W wrote:
>> > Thanks.  How do I activate the G1GC collector?  Do I do this by editing
>> a
>> > config file, or by adding a parameter when I start solr?
>> >
>> > Oracle's docs are pointing me to a file that supposedly is at
>> > instance-dir/OUD/config/java.properties, but I don't have that path.  I
>> am
>> > not sure what is meant by instance-dir here, but perhaps it means my JRE
>> > install, which is at
>> > /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre -- but
>> > there is no "OUD" directory in this location.
>>
>> The collector is chosen by the startup options given to Java, in this
>> case by the start script for Solr.  I've never heard of it being set by
>> a config in the JRE.
>>
>> In Solr 7, the start script defaults to the CMS collector.  We have
>> updated that to G1 in the latest Solr 8.x versions, because CMS has been
>> deprecated by Oracle.
>>
>> Adding the following lines to the correct solr.in.sh would change the
>> garbage collector to G1.  I got this from the "bin/solr" script in Solr
>> 8.5.1:
>>
>>GC_TUNE=('-XX:+UseG1GC' \
>>  '-XX:+PerfDisableSharedMem' \
>>  '-XX:+ParallelRefProcEnabled' \
>>  '-XX:MaxGCPauseMillis=250' \
>>  '-XX:+UseLargePages' \
>>  '-XX:+AlwaysPreTouch')
>>
>> If you used the service installer script to install Solr, then the
>> correct file to add this to is usually /etc/default/solr.in.sh ... but
>> if you did the install manually, it may be in the same bin directory
>> that contains the solr script itself.  Your initial message says the
>> solr home is /opt/solr/server/solr so I am assuming it's not running on
>> Windows.
>>
>> Thanks,
>> Shawn
>>
>

Analytics for Solr logs

[Zisis T.]

I've stumbled upon
https://github.com/apache/lucene-solr/blob/visual-guide/solr/solr-ref-guide/src/logs.adoc
which looks very interesting for getting insights into the Solr logs.

I cannot find though postlogs command inside the Solr bin dir (there is post
command though) nor a way to create the logs collection. I've looked into
solr-8.4.1 and solr-7.5.0 but could not find anything.

1) Is this still supported?
2) Where can I find the logs collection configuration? How can I create it?
3) Is post the same command as postlogs?

Thanks 



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Analytics for Solr logs

[Alexandre Rafalovitch]

The tool was introduced in Solr 8.5 and it is in bin/postlogs
location. It is quite new.

Regards,
   Alex.

On Tue, 13 Oct 2020 at 12:39, Zisis T.  wrote:
>
> I've stumbled upon
> https://github.com/apache/lucene-solr/blob/visual-guide/solr/solr-ref-guide/src/logs.adoc
> which looks very interesting for getting insights into the Solr logs.
>
> I cannot find though postlogs command inside the Solr bin dir (there is post
> command though) nor a way to create the logs collection. I've looked into
> solr-8.4.1 and solr-7.5.0 but could not find anything.
>
> 1) Is this still supported?
> 2) Where can I find the logs collection configuration? How can I create it?
> 3) Is post the same command as postlogs?
>
> Thanks
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: [CVE-2020-13957] The checks added to unauthenticated configset uploads in Apache Solr can be circumvented

[Tomás Fernández Löbbe]

Thanks Bernd, I missed 6.6.6 because it's not marked as a released version
in Jira. 6.6.6 is also affected.

On Mon, Oct 12, 2020 at 11:47 PM Bernd Fehling <
bernd.fehl...@uni-bielefeld.de> wrote:

> Good to know that Version 6.6.6 is not affected, so I am safe ;-)
>
> Regards
> Bernd
>
> Am 12.10.20 um 20:38 schrieb Tomas Fernandez Lobbe:
> > Severity: High
> >
> > Vendor: The Apache Software Foundation
> >
> > Versions Affected:
> > 6.6.0 to 6.6.5
> > 7.0.0 to 7.7.3
> > 8.0.0 to 8.6.2
> >
> > Description:
> > Solr prevents some features considered dangerous (which could be used for
> > remote code execution) to be configured in a ConfigSet that's uploaded
> via
> > API without authentication/authorization. The checks in place to prevent
> > such features can be circumvented by using a combination of UPLOAD/CREATE
> > actions.
> >
> > Mitigation:
> > Any of the following are enough to prevent this vulnerability:
> > * Disable UPLOAD command in ConfigSets API if not used by setting the
> > system property: "configset.upload.enabled" to "false" [1]
> > * Use Authentication/Authorization and make sure unknown requests aren't
> > allowed [2]
> > * Upgrade to Solr 8.6.3 or greater.
> > * If upgrading is not an option, consider applying the patch in
> SOLR-14663
> > ([3])
> > * No Solr API, including the Admin UI, is designed to be exposed to
> > non-trusted parties. Tune your firewall so that only trusted computers
> and
> > people are allowed access
> >
> > Credit:
> > Tomás Fernández Löbbe, András Salamon
> >
> > References:
> > [1] https://lucene.apache.org/solr/guide/8_6/configsets-api.html
> > [2]
> >
> https://lucene.apache.org/solr/guide/8_6/authentication-and-authorization-plugins.html
> > [3] https://issues.apache.org/jira/browse/SOLR-14663
> > [4] https://issues.apache.org/jira/browse/SOLR-14925
> > [5] https://wiki.apache.org/solr/SolrSecurity
> >
>

RE: Solr 7.7 - Few Questions

[Hanjan, Harinderdeep S.]

1. What tool they use to run Solr as a service on windows.
We use NSSM. https://nssm.cc/


-Original Message-
From: Manisha Rahatadkar [mailto:manisha.rahatad...@anjusoftware.com]
Sent: Tuesday, October 6, 2020 2:26 PM
To: solr-user@lucene.apache.org; ch...@opensourceconnections.com; Shawn Heisey 

Subject: [EXT] RE: Solr 7.7 - Few Questions

Hi All

First of all thanks to Shawn, Rahul and Charlie for taking time to reply my 
questions and valuable information.

I was very concerned about the size of the each document and on several follow 
ups got more information that the documents which have 0.5GB size are mp4 
documents and these are not synced to Solr.

@Shawn Heisey recommended NOT to use Windows because of windows license cost 
and service installer testing is done on Linux.
I agree with him. We are using NSSM tool to run solr as a service.

Are there any members here using Solr on Windows? I look forward to hear from 
them on:

1. What tool they use to run Solr as a service on windows.
2. How to set up the disaster recovery?
3. How to scale up the servers for the better performance?

Thanks in advance and looking forward to hear back your experiences on Solr 
Scale up.

Regards,
Manisha Rahatadkar

-Original Message-
From: Rahul Goswami 
Sent: Sunday, October 4, 2020 11:49 PM
To: ch...@opensourceconnections.com; solr-user@lucene.apache.org
Subject: Re: Solr 7.7 - Few Questions

Charlie,
Thanks for providing an alternate approach to doing this. It would be 
interesting to know how one  could go about organizing the docs in this case? 
(Nested documents?) How would join queries perform on a large
index(200 million+ docs)?

Thanks,
Rahul



On Fri, Oct 2, 2020 at 5:55 AM Charlie Hull  wrote:

> Hi Rahul,
>
>
>
> In addition to the wise advice below: remember in Solr, a 'document'
> is
>
> just the name for the thing that would appear as one of the results
> when
>
> you search (analagous to a database record). It's not the same
>
> conceptually as a 'Word document' or a 'PDF document'. If your source
>
> documents are so big, consider how they might be broken into parts, or
>
> whether you really need to index all of them for retrieval purposes,
> or
>
> what parts of them need to be extracted as text. Thus, the Solr
>
> documents don't necessarily need to be as large as your source documents.
>
>
>
> Consider an email size 20kb with ten PDF attachments, each 20MB. You
>
> probably shouldn't push all this data into a single Solr document, but
>
> you *could* index them as 11 separate Solr documents, but with
> metadata
>
> to indicate that one is an email and ten are PDFs, and a shared ID of
>
> some kind to indicate they're related. Then at query time there are
>
> various ways for you to group these together, so for example if the
>
> query hit one of the PDFs you could show the user the original email,
>
> plus the 9 other attachments, using the shared ID as a key.
>
>
>
> HTH,
>
>
>
> Charlie
>
>
>
> On 02/10/2020 01:53, Rahul Goswami wrote:
>
> > Manisha,
>
> > In addition to what Shawn has mentioned above, I would also like you
> > to
>
> > reevaluate your use case. Do you *need to* index the whole document ? eg:
>
> > If it's an email, the body of the email *might* be more important
> > than
> any
>
> > attachments, in which case you could choose to only index the email
> > body
>
> > and ignore (or only partially index) the text from attachments. If
> > you
>
> > could afford to index the documents partially, you could consider
> > Solr's
>
> > "Limit token count filter": See the link below.
>
> >
>
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org
> _solr_guide_7-5F7_filter-2Ddescriptions.html-23limi&d=DwIGaQ&c=jdm1Hby
> _BzoqwoYzPsUCHSCnNps9LuidNkyKDuvdq3M&r=K2RffikYjYEm9pNz7rnNO_vxytl-lju
> jndRmklLfE1I&m=2ozKDmMVWaDgCqsPaYwwEELoGjA5d6xC9xgH28tiErs&s=ysHd67CYE
> hPBEEWIda8ItM0R5gllayaddUyTwqm0glw&e=
> t-token-count-filter
>
> >
>
> > You'll need to configure it in the schema for the "index" analyzer
> > for
> the
>
> > data type of the field with large text.
>
> > Indexing documents of the order of half a GB will definitely come to
> > hurt
>
> > your operations, if not now, later (think OOM, extremely slow atomic
>
> > updates, long running merges etc.).
>
> >
>
> > - Rahul
>
> >
>
> >
>
> >
>
> > On Thu, Oct 1, 2020 at 7:06 PM Shawn Heisey  wrote:
>
> >
>
> >> On 10/1/2020 6:57 AM, Manisha Rahatadkar wrote:
>
> >>> We are using Apache Solr 7.7 on Windows platform. The data is
> >>> synced to
>
> >> Solr using Solr.Net commit. The data is being synced to SOLR in batches.
>
> >> The document size is very huge (~0.5GB average) and solr indexing
> >> is
> taking
>
> >> long time. Total document size is ~200GB. As the solr commit is
> >> done as
> a
>
> >> part of API, the API calls are failing as document indexing is not
>
> >> completed.
>
> >>
>
> >> A single document is five hundred megabytes?  What kind of
> >> documents do
>
> >> you have?  You 

Re: Memory line in status output

[Ryan W]

I think I have it sorted. At this point I'm using GCG1, I take it, because
most recently I started Solr as a service...

service solr start

And that is running solr by way of /etc/init.d/solr because I don't have
any systemd unit for solr, as explained here...
https://askubuntu.com/questions/903354/difference-between-systemctl-and-service-commands

And I can see in the System V script for solr that /etc/default/solr.in.sh
is the relevant config file.


On Tue, Oct 13, 2020 at 11:23 AM Ryan W  wrote:

> Or, perhaps if I start solr like so
>
> service solr start
>
> ...it will use the solr.in.sh at /etc/default/solr.in.sh ?
>
>
>
> On Tue, Oct 13, 2020 at 11:19 AM Ryan W  wrote:
>
>> This is how I start solr:
>>
>> /opt/solr/bin/solr start
>>
>> In my /etc/default/solr.in.sh, I have this...
>>
>> GC_TUNE=" \
>> -XX:+UseG1GC \
>> -XX:+ParallelRefProcEnabled \
>> -XX:G1HeapRegionSize=8m \
>> -XX:MaxGCPauseMillis=200 \
>> -XX:+UseLargePages \
>> -XX:+AggressiveOpts \
>> "
>>
>> But I don't know how to tell if Solr is using that file.
>>
>> In my /opt/solr/bin there is no solr.in.sh, but there is a
>> solr.in.sh.orig -- perhaps I should copy my /etc/default/solr.in.sh to
>> /opt/solr/bin ?
>>
>> I am running Linux (RHEL).  The Solr version is 7.7.2.  Solr 8.x is not
>> compatible with my application.
>>
>> Thank you.
>>
>>
>> On Mon, Oct 12, 2020 at 9:46 PM Shawn Heisey 
>> wrote:
>>
>>> On 10/12/2020 5:11 PM, Ryan W wrote:
>>> > Thanks.  How do I activate the G1GC collector?  Do I do this by
>>> editing a
>>> > config file, or by adding a parameter when I start solr?
>>> >
>>> > Oracle's docs are pointing me to a file that supposedly is at
>>> > instance-dir/OUD/config/java.properties, but I don't have that path.
>>> I am
>>> > not sure what is meant by instance-dir here, but perhaps it means my
>>> JRE
>>> > install, which is at
>>> > /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre -- but
>>> > there is no "OUD" directory in this location.
>>>
>>> The collector is chosen by the startup options given to Java, in this
>>> case by the start script for Solr.  I've never heard of it being set by
>>> a config in the JRE.
>>>
>>> In Solr 7, the start script defaults to the CMS collector.  We have
>>> updated that to G1 in the latest Solr 8.x versions, because CMS has been
>>> deprecated by Oracle.
>>>
>>> Adding the following lines to the correct solr.in.sh would change the
>>> garbage collector to G1.  I got this from the "bin/solr" script in Solr
>>> 8.5.1:
>>>
>>>GC_TUNE=('-XX:+UseG1GC' \
>>>  '-XX:+PerfDisableSharedMem' \
>>>  '-XX:+ParallelRefProcEnabled' \
>>>  '-XX:MaxGCPauseMillis=250' \
>>>  '-XX:+UseLargePages' \
>>>  '-XX:+AlwaysPreTouch')
>>>
>>> If you used the service installer script to install Solr, then the
>>> correct file to add this to is usually /etc/default/solr.in.sh ... but
>>> if you did the install manually, it may be in the same bin directory
>>> that contains the solr script itself.  Your initial message says the
>>> solr home is /opt/solr/server/solr so I am assuming it's not running on
>>> Windows.
>>>
>>> Thanks,
>>> Shawn
>>>
>>

Re: Memory line in status output

[Walter Underwood]

The home page of the Solr admin UI shows all of the options to the JVM.
That will include the choice of garbage collector.

You can also see the options with “ps -ef | grep solr”.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Oct 13, 2020, at 1:19 PM, Ryan W  wrote:
> 
> I think I have it sorted. At this point I'm using GCG1, I take it, because
> most recently I started Solr as a service...
> 
> service solr start
> 
> And that is running solr by way of /etc/init.d/solr because I don't have
> any systemd unit for solr, as explained here...
> https://askubuntu.com/questions/903354/difference-between-systemctl-and-service-commands
> 
> And I can see in the System V script for solr that /etc/default/solr.in.sh
> is the relevant config file.
> 
> 
> On Tue, Oct 13, 2020 at 11:23 AM Ryan W  wrote:
> 
>> Or, perhaps if I start solr like so
>> 
>> service solr start
>> 
>> ...it will use the solr.in.sh at /etc/default/solr.in.sh ?
>> 
>> 
>> 
>> On Tue, Oct 13, 2020 at 11:19 AM Ryan W  wrote:
>> 
>>> This is how I start solr:
>>> 
>>> /opt/solr/bin/solr start
>>> 
>>> In my /etc/default/solr.in.sh, I have this...
>>> 
>>> GC_TUNE=" \
>>> -XX:+UseG1GC \
>>> -XX:+ParallelRefProcEnabled \
>>> -XX:G1HeapRegionSize=8m \
>>> -XX:MaxGCPauseMillis=200 \
>>> -XX:+UseLargePages \
>>> -XX:+AggressiveOpts \
>>> "
>>> 
>>> But I don't know how to tell if Solr is using that file.
>>> 
>>> In my /opt/solr/bin there is no solr.in.sh, but there is a
>>> solr.in.sh.orig -- perhaps I should copy my /etc/default/solr.in.sh to
>>> /opt/solr/bin ?
>>> 
>>> I am running Linux (RHEL).  The Solr version is 7.7.2.  Solr 8.x is not
>>> compatible with my application.
>>> 
>>> Thank you.
>>> 
>>> 
>>> On Mon, Oct 12, 2020 at 9:46 PM Shawn Heisey 
>>> wrote:
>>> 
 On 10/12/2020 5:11 PM, Ryan W wrote:
> Thanks.  How do I activate the G1GC collector?  Do I do this by
 editing a
> config file, or by adding a parameter when I start solr?
> 
> Oracle's docs are pointing me to a file that supposedly is at
> instance-dir/OUD/config/java.properties, but I don't have that path.
 I am
> not sure what is meant by instance-dir here, but perhaps it means my
 JRE
> install, which is at
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre -- but
> there is no "OUD" directory in this location.
 
 The collector is chosen by the startup options given to Java, in this
 case by the start script for Solr.  I've never heard of it being set by
 a config in the JRE.
 
 In Solr 7, the start script defaults to the CMS collector.  We have
 updated that to G1 in the latest Solr 8.x versions, because CMS has been
 deprecated by Oracle.
 
 Adding the following lines to the correct solr.in.sh would change the
 garbage collector to G1.  I got this from the "bin/solr" script in Solr
 8.5.1:
 
   GC_TUNE=('-XX:+UseG1GC' \
 '-XX:+PerfDisableSharedMem' \
 '-XX:+ParallelRefProcEnabled' \
 '-XX:MaxGCPauseMillis=250' \
 '-XX:+UseLargePages' \
 '-XX:+AlwaysPreTouch')
 
 If you used the service installer script to install Solr, then the
 correct file to add this to is usually /etc/default/solr.in.sh ... but
 if you did the install manually, it may be in the same bin directory
 that contains the solr script itself.  Your initial message says the
 solr home is /opt/solr/server/solr so I am assuming it's not running on
 Windows.
 
 Thanks,
 Shawn
 
>>> 

Re: Memory line in status output

[Ryan W]

Thanks.  The G1 docs say "G1 is designed to provide good overall
performance without the need to specify additional options."

Would that look like this...

GC_TUNE=" \
-XX:+UseG1GC \
"

Is that the most minimal config? Is it typical to use it without options?

On Tue, Oct 13, 2020 at 4:22 PM Walter Underwood 
wrote:

> The home page of the Solr admin UI shows all of the options to the JVM.
> That will include the choice of garbage collector.
>
> You can also see the options with “ps -ef | grep solr”.
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
> > On Oct 13, 2020, at 1:19 PM, Ryan W  wrote:
> >
> > I think I have it sorted. At this point I'm using GCG1, I take it,
> because
> > most recently I started Solr as a service...
> >
> > service solr start
> >
> > And that is running solr by way of /etc/init.d/solr because I don't have
> > any systemd unit for solr, as explained here...
> >
> https://askubuntu.com/questions/903354/difference-between-systemctl-and-service-commands
> >
> > And I can see in the System V script for solr that /etc/default/
> solr.in.sh
> > is the relevant config file.
> >
> >
> > On Tue, Oct 13, 2020 at 11:23 AM Ryan W  wrote:
> >
> >> Or, perhaps if I start solr like so
> >>
> >> service solr start
> >>
> >> ...it will use the solr.in.sh at /etc/default/solr.in.sh ?
> >>
> >>
> >>
> >> On Tue, Oct 13, 2020 at 11:19 AM Ryan W  wrote:
> >>
> >>> This is how I start solr:
> >>>
> >>> /opt/solr/bin/solr start
> >>>
> >>> In my /etc/default/solr.in.sh, I have this...
> >>>
> >>> GC_TUNE=" \
> >>> -XX:+UseG1GC \
> >>> -XX:+ParallelRefProcEnabled \
> >>> -XX:G1HeapRegionSize=8m \
> >>> -XX:MaxGCPauseMillis=200 \
> >>> -XX:+UseLargePages \
> >>> -XX:+AggressiveOpts \
> >>> "
> >>>
> >>> But I don't know how to tell if Solr is using that file.
> >>>
> >>> In my /opt/solr/bin there is no solr.in.sh, but there is a
> >>> solr.in.sh.orig -- perhaps I should copy my /etc/default/solr.in.sh to
> >>> /opt/solr/bin ?
> >>>
> >>> I am running Linux (RHEL).  The Solr version is 7.7.2.  Solr 8.x is not
> >>> compatible with my application.
> >>>
> >>> Thank you.
> >>>
> >>>
> >>> On Mon, Oct 12, 2020 at 9:46 PM Shawn Heisey 
> >>> wrote:
> >>>
>  On 10/12/2020 5:11 PM, Ryan W wrote:
> > Thanks.  How do I activate the G1GC collector?  Do I do this by
>  editing a
> > config file, or by adding a parameter when I start solr?
> >
> > Oracle's docs are pointing me to a file that supposedly is at
> > instance-dir/OUD/config/java.properties, but I don't have that path.
>  I am
> > not sure what is meant by instance-dir here, but perhaps it means my
>  JRE
> > install, which is at
> > /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre --
> but
> > there is no "OUD" directory in this location.
> 
>  The collector is chosen by the startup options given to Java, in this
>  case by the start script for Solr.  I've never heard of it being set
> by
>  a config in the JRE.
> 
>  In Solr 7, the start script defaults to the CMS collector.  We have
>  updated that to G1 in the latest Solr 8.x versions, because CMS has
> been
>  deprecated by Oracle.
> 
>  Adding the following lines to the correct solr.in.sh would change the
>  garbage collector to G1.  I got this from the "bin/solr" script in
> Solr
>  8.5.1:
> 
>    GC_TUNE=('-XX:+UseG1GC' \
>  '-XX:+PerfDisableSharedMem' \
>  '-XX:+ParallelRefProcEnabled' \
>  '-XX:MaxGCPauseMillis=250' \
>  '-XX:+UseLargePages' \
>  '-XX:+AlwaysPreTouch')
> 
>  If you used the service installer script to install Solr, then the
>  correct file to add this to is usually /etc/default/solr.in.sh ...
> but
>  if you did the install manually, it may be in the same bin directory
>  that contains the solr script itself.  Your initial message says the
>  solr home is /opt/solr/server/solr so I am assuming it's not running
> on
>  Windows.
> 
>  Thanks,
>  Shawn
> 
> >>>
>
>

Re: Memory line in status output

[Walter Underwood]

I recommend using the options mentioned in recent messages on this list.

Solr has pretty specific memory demands, with lots of allocations with a
lifetime of a single request, plus very long-lived allocations that aren’t freed
until they are evicted from a cache.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Oct 13, 2020, at 1:35 PM, Ryan W  wrote:
> 
> Thanks.  The G1 docs say "G1 is designed to provide good overall
> performance without the need to specify additional options."
> 
> Would that look like this...
> 
> GC_TUNE=" \
> -XX:+UseG1GC \
> "
> 
> Is that the most minimal config? Is it typical to use it without options?
> 
> On Tue, Oct 13, 2020 at 4:22 PM Walter Underwood 
> wrote:
> 
>> The home page of the Solr admin UI shows all of the options to the JVM.
>> That will include the choice of garbage collector.
>> 
>> You can also see the options with “ps -ef | grep solr”.
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>>> On Oct 13, 2020, at 1:19 PM, Ryan W  wrote:
>>> 
>>> I think I have it sorted. At this point I'm using GCG1, I take it,
>> because
>>> most recently I started Solr as a service...
>>> 
>>> service solr start
>>> 
>>> And that is running solr by way of /etc/init.d/solr because I don't have
>>> any systemd unit for solr, as explained here...
>>> 
>> https://askubuntu.com/questions/903354/difference-between-systemctl-and-service-commands
>>> 
>>> And I can see in the System V script for solr that /etc/default/
>> solr.in.sh
>>> is the relevant config file.
>>> 
>>> 
>>> On Tue, Oct 13, 2020 at 11:23 AM Ryan W  wrote:
>>> 
 Or, perhaps if I start solr like so
 
 service solr start
 
 ...it will use the solr.in.sh at /etc/default/solr.in.sh ?
 
 
 
 On Tue, Oct 13, 2020 at 11:19 AM Ryan W  wrote:
 
> This is how I start solr:
> 
> /opt/solr/bin/solr start
> 
> In my /etc/default/solr.in.sh, I have this...
> 
> GC_TUNE=" \
> -XX:+UseG1GC \
> -XX:+ParallelRefProcEnabled \
> -XX:G1HeapRegionSize=8m \
> -XX:MaxGCPauseMillis=200 \
> -XX:+UseLargePages \
> -XX:+AggressiveOpts \
> "
> 
> But I don't know how to tell if Solr is using that file.
> 
> In my /opt/solr/bin there is no solr.in.sh, but there is a
> solr.in.sh.orig -- perhaps I should copy my /etc/default/solr.in.sh to
> /opt/solr/bin ?
> 
> I am running Linux (RHEL).  The Solr version is 7.7.2.  Solr 8.x is not
> compatible with my application.
> 
> Thank you.
> 
> 
> On Mon, Oct 12, 2020 at 9:46 PM Shawn Heisey 
> wrote:
> 
>> On 10/12/2020 5:11 PM, Ryan W wrote:
>>> Thanks.  How do I activate the G1GC collector?  Do I do this by
>> editing a
>>> config file, or by adding a parameter when I start solr?
>>> 
>>> Oracle's docs are pointing me to a file that supposedly is at
>>> instance-dir/OUD/config/java.properties, but I don't have that path.
>> I am
>>> not sure what is meant by instance-dir here, but perhaps it means my
>> JRE
>>> install, which is at
>>> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre --
>> but
>>> there is no "OUD" directory in this location.
>> 
>> The collector is chosen by the startup options given to Java, in this
>> case by the start script for Solr.  I've never heard of it being set
>> by
>> a config in the JRE.
>> 
>> In Solr 7, the start script defaults to the CMS collector.  We have
>> updated that to G1 in the latest Solr 8.x versions, because CMS has
>> been
>> deprecated by Oracle.
>> 
>> Adding the following lines to the correct solr.in.sh would change the
>> garbage collector to G1.  I got this from the "bin/solr" script in
>> Solr
>> 8.5.1:
>> 
>>  GC_TUNE=('-XX:+UseG1GC' \
>>'-XX:+PerfDisableSharedMem' \
>>'-XX:+ParallelRefProcEnabled' \
>>'-XX:MaxGCPauseMillis=250' \
>>'-XX:+UseLargePages' \
>>'-XX:+AlwaysPreTouch')
>> 
>> If you used the service installer script to install Solr, then the
>> correct file to add this to is usually /etc/default/solr.in.sh ...
>> but
>> if you did the install manually, it may be in the same bin directory
>> that contains the solr script itself.  Your initial message says the
>> solr home is /opt/solr/server/solr so I am assuming it's not running
>> on
>> Windows.
>> 
>> Thanks,
>> Shawn
>> 
> 
>> 
>> 

Need help in trying to understand the error

[yaswanth kumar]

I am seeing the below errors frequently on the solr logs, every
functionality seems to be working fine but not really sure why there are
lots of these errors happening in the backend

Using : solr8.2, zoo 3.4
we have enable solr basicauthentication with security.json

2020-10-13 20:37:12.320 ERROR (qtp969996005-4438) [   ]
o.a.s.c.s.i.HttpClientUtil  => org.apache.solr.common.SolrException:
javax.crypto.BadPaddingException: RSA private key operation failed
at org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:325)
org.apache.solr.common.SolrException: javax.crypto.BadPaddingException: RSA
private key operation failed
at org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys.java:325)
~[solr-core-8.2.0.jar:8.2.0 31d7ec7bbfdcd2c4cc61d9d35e962165410b65fe -
ivera - 2019-07-19 15:11:04]
at
org.apache.solr.security.PKIAuthenticationPlugin.generateToken(PKIAuthenticationPlugin.java:305)
~[solr-core-8.2.0.jar:8.2.0 31d7ec7bbfdcd2c4cc61d9d35e962165410b65fe -
ivera - 2019-07-19 15:11:04]
at
org.apache.solr.security.PKIAuthenticationPlugin.setHeader(PKIAuthenticationPlugin.java:311)
~[solr-core-8.2.0.jar:8.2.0 31d7ec7bbfdcd2c4cc61d9d35e962165410b65fe -
ivera - 2019-07-19 15:11:04]
at
org.apache.solr.security.PKIAuthenticationPlugin$HttpHeaderClientInterceptor.process(PKIAuthenticationPlugin.java:271)
~[solr-core-8.2.0.jar:8.2.0 31d7ec7bbfdcd2c4cc61d9d35e962165410b65fe -
ivera - 2019-07-19 15:11:04]
at
org.apache.solr.client.solrj.impl.HttpClientUtil$DynamicInterceptor$1.accept(HttpClientUtil.java:179)
~[solr-solrj-8.2.0.jar:8.2.0 31d7ec7bbfdcd2c4cc61d9d35e962165410b65fe -
ivera - 2019-07-19 15:11:07]
at
org.apache.solr.client.solrj.impl.HttpClientUtil$DynamicInterceptor$1.accept(HttpClientUtil.java:174)
~[solr-solrj-8.2.0.jar:8.2.0 31d7ec7bbfdcd2c4cc61d9d35e962165410b65fe -
ivera - 2019-07-19 15:11:07]
at
java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:804)
~[?:?]
at
org.apache.solr.client.solrj.impl.HttpClientUtil$DynamicInterceptor.process(HttpClientUtil.java:174)
~[solr-solrj-8.2.0.jar:8.2.0 31d7ec7bbfdcd2c4cc61d9d35e962165410b65fe -
ivera - 2019-07-19 15:11:07]
at
org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:133)
~[httpcore-4.4.10.jar:4.4.10]
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:183)
~[httpclient-4.5.6.jar:4.5.6]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
~[httpclient-4.5.6.jar:4.5.6]
at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:688)
~[solr-core-8.2.0.jar:8.2.0 31d7ec7bbfdcd2c4cc61d9d35e962165410b65fe -
ivera - 2019-07-19 15:11:04]
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:550)
~[solr-core-8.2.0.jar:8.2.0 31d7ec7bbfdcd2c4cc61d9d35e962165410b65fe -
ivera - 2019-07-19 15:11:04]
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)
~[solr-core-8.2.0.jar:8.2.0 31d7ec7bbfdcd2c4cc61d9d35e962165410b65fe -
ivera - 2019-07-19 15:11:04]
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
~[solr-core-8.2.0.jar:8.2.0 31d7ec7bbfdcd2c4cc61d9d35e962165410b65fe -
ivera - 2019-07-19 15:11:04]
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
~[jetty-security-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
~[jetty-server-9.4.19.v2

Strange fetch streaming expression doesn't fetch fields sometimes?

[uyilmaz]

Hi all,

I have a streaming expression looking like:

fetch(
  myAlias,
  top(
n=3,
  various expressions here
sort="count(*) desc"
  ),
  fl="username", on="userid=userid", batchSize=3
)

which fails to fetch username field for the 1st result:

{
 "result-set":{
  "docs":[{
"userid":"123123",
"count(*)":58}
   ,{
"userid":"123123123",
"count(*)":32,
"username":"Ayha"}
   ,{
"userid":"12432423321323",
"count(*)":30,
"username":"MEHM"}
   ,{
"EOF":true,
"RESPONSE_TIME":34889}]}}

But strangely, when I change n and batchSize both to 2 and touch nothing else, 
fetch fetches the first username correctly:

fetch(
  myAlias,
  top(
n=2,
  various expressions here
sort="count(*) desc"
  ),
  fl="username", on="userid=userid", batchSize=2
)

Result is:

{
 "result-set":{
  "docs":[{
"userid":"123123",
"count(*)":58,
"username":"mura"}
   ,{
"userid":"123123123",
"count(*)":32,
"username":"Ayha"}
   ,{
"EOF":true,
"RESPONSE_TIME":34889}]}}

What can be the problem?

Regards

~~ufuk

-- 
uyilmaz 

unable to addReplica

[yaswanth kumar]

Can someone help me in fixing the below error that I am seeing when trying
to use addReplica command

curl -k -u username:password "
https://x.x.x.x:8080/solr/admin/collections?action=ADDREPLICA&collection=Test&shard=shard1
"

Solr version: 8.2
Zookeeper: 3.4

Solr BasicAuthentication was enabled

Error seeing when trying to hit the above
 "responseHeader":{
"status":500,
"QTime":152},
  "failure":{

"9.37.195.125:8080_solr":"org.apache.solr.common.SolrException:javax.crypto.BadPaddingException:
RSA private key operation failed"},
  "Operation addreplica caused
exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
ADDREPLICA failed to create replica",
  "exception":{
"msg":"ADDREPLICA failed to create replica",
"rspCode":500},
  "error":{
"metadata":[
  "error-class","org.apache.solr.common.SolrException",
  "root-error-class","org.apache.solr.common.SolrException"],
"msg":"ADDREPLICA failed to create replica",
"trace":"org.apache.solr.common.SolrException: ADDREPLICA failed to
create replica\n\tat
org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:280)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:252)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:820)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:786)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:505)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427)\n\tat
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321)\n\tat
org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)\n\tat
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)\n\tat
org.eclipse.je

Re: Strange fetch streaming expression doesn't fetch fields sometimes?

[uyilmaz]

I think I found the reason right after asking (facepalm), but it took me days 
to realize this.

I think fetch performs a naive "in" query, something like:

q="userid:(123123 123123123 12432423321323)&rows={batchSize}"

When userid to document relation is one-to-many, it is possible that above 
query will result in documents consisting entirely of last two userid's 
documents, so the first one is left out, resulting in empty username. Docs 
state that one to many is not supported with fetch, but I didn't stumble onto 
this issue until recently so I just assumed it would work.

Sorry to take your time, I hope this helps somebody later.

Have a nice day.

On Wed, 14 Oct 2020 00:38:05 +0300
uyilmaz  wrote:

> 
> Hi all,
> 
> I have a streaming expression looking like:
> 
> fetch(
>   myAlias,
>   top(
>   n=3,
>   various expressions here
> sort="count(*) desc"
>   ),
>   fl="username", on="userid=userid", batchSize=3
> )
> 
> which fails to fetch username field for the 1st result:
> 
> {
>  "result-set":{
>   "docs":[{
> "userid":"123123",
> "count(*)":58}
>,{
> "userid":"123123123",
> "count(*)":32,
> "username":"Ayha"}
>,{
> "userid":"12432423321323",
> "count(*)":30,
> "username":"MEHM"}
>,{
> "EOF":true,
> "RESPONSE_TIME":34889}]}}
>   
> But strangely, when I change n and batchSize both to 2 and touch nothing 
> else, fetch fetches the first username correctly:
> 
> fetch(
>   myAlias,
>   top(
>   n=2,
>   various expressions here
> sort="count(*) desc"
>   ),
>   fl="username", on="userid=userid", batchSize=2
> )
> 
> Result is:
> 
> {
>  "result-set":{
>   "docs":[{
> "userid":"123123",
> "count(*)":58,
> "username":"mura"}
>,{
> "userid":"123123123",
> "count(*)":32,
> "username":"Ayha"}
>,{
> "EOF":true,
> "RESPONSE_TIME":34889}]}}
>   
> What can be the problem?
> 
> Regards
> 
> ~~ufuk
> 
> -- 
> uyilmaz 


-- 
uyilmaz 

Need urgent help -- High cpu on solr

[yaswanth kumar]

I am using solr 8.2 with zoo 3.4, and configured 5 node solr cloud with
around 100 collections each collection having ~20k documents.

These nodes are vm's with 6 core cpu and 2 cores per socket. All of sudden
seeing hikes on CPU's and which brought down some nodes (GONE state on solr
cloud and also faced latencies while trying to login to those nodes ssh)

Memory : 32GB and 20GB was allotted for jvm heap on solr config.

 
 
 
 200
100
true
 false
4

These are just from the defaults that shipped with SOLR package.

One data point is that these nodes gets very frequent hits to them for
searching, so do I need to consider increasing the above sizes to get down
the CPU's and see more stable solr cloud?

-- 
Thanks & Regards,
Yaswanth Kumar Konathala.
yaswanth...@gmail.com

Is metrics api enabled by default in solr 8.2

[yaswanth kumar]

Can I get some info on where to disable or enable metrics api on solr 8.2 ?

I believe its enabled by default on solr 8.2 , where can I check the
configurations? and also how can I disable if I want to disable it

-- 
Thanks & Regards,
Yaswanth Kumar Konathala.
yaswanth...@gmail.com