Segments with 500KB size (source: "flush") in a cluster with ramBufferSize of 2GB

[Daniel Ortega]

Hi Guys,

Could someone explain me why I have segments of 500 KB (with source "flush"
and only with 91 documents) if I have a ramBufferSizeMB of 2GB
and maxBufferedDocs not definined?

Thanks in advance,
Daniel

Re: Segments with 500KB size (source: "flush") in a cluster with ramBufferSize of 2GB

[Daniel Ortega]

Hi Erick,

1) maxDocs setting is not defined in our config. As far as we know there
isn't a default value.

2) ramBufferSizeMB cannot be exceeded because written segment size are less
than 1 MB and we have defined a maximum of 2 GB.

3) The autocommit interval is 15 minutes. This could be the reason but it's
weird because we are updating more than one thousand documents per second
(across the cluster).

4) We disabled explicit commits from clients :)

5) We're using SolrJ but with commitWithin option not defined.

Anyway thanks for your reply!

2017-07-06 16:56 GMT+02:00 Erick Erickson :

> You'll get a new segment whenever your time expires as well. Segments
> are created whenever the _first_ of
> 1> maxDocs exceeded
> 2> ramBufferSizeMB is exceded
> 3> the autocommit _time_ interval is exceeded
> 4> someone does a commit
> 5> if someone is indexing via SolrJ and specifies a "commitWithin" for
> one of the calls.
>
> Can you guarantee that none of these occur?
>
> Best,
> Erick
>
> On Thu, Jul 6, 2017 at 6:14 AM, Daniel Ortega
>  wrote:
> > Hi Guys,
> >
> > Could someone explain me why I have segments of 500 KB (with source
> "flush"
> > and only with 91 documents) if I have a ramBufferSizeMB of 2GB
> > and maxBufferedDocs not definined?
> >
> > Thanks in advance,
> > Daniel
>

Excessive resources consumption migrating from Solr 6.6.0 Master/Slave to SolrCloud 6.6.0 (dozen times more resources)

[Daniel Ortega]

*Main Problems*


We are involved in a migration from Solr Master/Slave infrastructure to
SolrCloud infrastructure.



The main problems that we have now are:



   - Excessive resources consumption: Currently we have 5 instances with 80
   processors/768 GB RAM each instance using SSD Hard Disk Drives that doesn't
   support the load that we have in the other architecture. In our
   Master-Slave architecture we have only 7 Virtual Machines with lower specs
   (4 processors and 16 GB each instance using SSD Hard Disk Drives too). So,
   at the moment our SolrCloud infrastructure is wasting several dozen times
   more resources than our Solr Master/Slave infrastructure.
   - Despite spending more resources we have worst query times (compared to
   Solr in master/slave architecture)


*Search infrastructure (SolrCloud infrastructure)*



As we cannot use DIH Handler (which is what we use in Solr Master/Slave), we
have developed an application which reads every transaction from Oracle,
builds a document collection searching in the database and sends the result
to the */update* handler every 200 milliseconds using SolrJ client. This
application tries to delete the possible duplicates in each update window,
but we are using solr’s de-duplication techniques

 too.



We are indexing ~100 documents per second (with peaks of ~1000 documents
per second).



Every search query is centralized in other application which exposes a DSL
behind a REST API and uses SolrJ client too to perform queries. We have
peaks of 2000 QPS.

*Cluster structure **(SolrCloud infrastructure)*



At the moment, the cluster has 30 SolrCloud instances with the same specs
(Same physical hosts, same JVM Settings, etc.).



*Main collection*



In our use case we are using this collection as a NoSQL database basically.
Our document is composed of about 300 fields that represents an advert, and
is a denormalization of its relational representation in Oracle.


We are using all our nodes to store the  collection in 3 shards. So, each
shard has 10 replicas.


At the moment, we are only indexing a subset of the adverts stored in
Oracle, but our goal is to store all the ads that we have in the DB (a few
tens of millions of documents). We have NRT requirements, so we need to
index every document as soon as posible once it’s changed in Oracle.



We have defined the properties of each field (if it’s stored/indexed or
not, if should be defined as DocValue, etc…) considering the use of that
field.



*Index size **(SolrCloud infrastructure)*



The index size is currently above 6 GB, storing 1.300.000 documents in each
shard. So, we are storing 3.900.000 documents and the total index size is
18 GB.



*Indexation **(SolrCloud infrastructure)*



The commits *aren’t* triggered by the application described before. The
hardcommit/softcommit interval are configured in Solr:



   - *HardCommit:* every 15 minutes (with opensearcher = false)
   - *SoftCommit:* every 5 seconds



*Apache Solr Version*



We are currently using the last version of Solr (6.6.0) under an Oracle VM
(Java(TM) SE Runtime Environment (build 1.8.0_131-b11) Oracle (64 bits)) in
both deployments.


The question is... What is wrong here?!?!?!

Re: Excessive resources consumption migrating from Solr 6.6.0 Master/Slave to SolrCloud 6.6.0 (dozen times more resources)

[Daniel Ortega]

Hi Scott,

- *Can you describe the process that queries the DB and sends records to *
*Solr?*

We are enqueueing ids during every ORACLE transaction (in insert/updates).

An application dequeues every id and perform queries against dozen of
tables in the relational model to retrieve the fields to build the
document.  As we know that we are modifying the same ORACLE row in
different (but consecutive) transactions, we store only the last version of
the modified documents in a map data structure.

The application has a configurable interval to send the documents stored in
the map to the update handler (we have tested different intervals from few
milliseconds to several seconds) using the SolrJ client. Actually we are
sending all the documents every 15 seconds.

This application is developed using Java, Spring and Maven and we have
several instances.

-* Is it a SolrJ-based application?*

Yes, it is. We aren't using the last version of SolrJ client (we are
currently using SolrJ v6.3.0).

- *If it is, which client package are you using?*

I don't know exactly what do you mean saying 'client package' :)

- *How many documents do you send at once?*

It depends on the defined interval described before and the number of
transactions executed in our relational database. From dozens to few
hundreds (and even thousands).

- *Are you sending your indexing or query traffic through a load balancer?*

We aren't using a load balancer for indexing, but we have all our Rest
Query services through an HAProxy (using 'leastconn' algorithm). The Rest
Query Services performs queries using the CloudSolrClient.

Thanks for your reply,
if you need any further information don't hesitate to ask

Daniel

2017-08-23 14:57 GMT+02:00 Scott Stults :

> Hi Daniel,
>
> Great background information about your setup! I've got just a few more
> questions:
>
> - Can you describe the process that queries the DB and sends records to
> Solr?
> - Is it a SolrJ-based application?
> - If it is, which client package are you using?
> - How many documents do you send at once?
> - Are you sending your indexing or query traffic through a load balancer?
>
> If you're sending documents to each replica as fast as they can take them,
> you might be seeing a bottleneck at the shard leaders. The SolrJ
> CloudSolrClient finds out from Zookeeper which nodes are the shard leaders
> and sends docs directly to them.
>
>
> -Scott
>
> On Tue, Aug 22, 2017 at 2:16 PM, Daniel Ortega <
> danielortegauf...@gmail.com>
> wrote:
>
> > *Main Problems*
> >
> >
> > We are involved in a migration from Solr Master/Slave infrastructure to
> > SolrCloud infrastructure.
> >
> >
> >
> > The main problems that we have now are:
> >
> >
> >
> >- Excessive resources consumption: Currently we have 5 instances with
> 80
> >processors/768 GB RAM each instance using SSD Hard Disk Drives that
> > doesn't
> >support the load that we have in the other architecture. In our
> >Master-Slave architecture we have only 7 Virtual Machines with lower
> > specs
> >(4 processors and 16 GB each instance using SSD Hard Disk Drives too).
> > So,
> >at the moment our SolrCloud infrastructure is wasting several dozen
> > times
> >more resources than our Solr Master/Slave infrastructure.
> >- Despite spending more resources we have worst query times (compared
> to
> >Solr in master/slave architecture)
> >
> >
> > *Search infrastructure (SolrCloud infrastructure)*
> >
> >
> >
> > As we cannot use DIH Handler (which is what we use in Solr Master/Slave),
> > we
> > have developed an application which reads every transaction from Oracle,
> > builds a document collection searching in the database and sends the
> result
> > to the */update* handler every 200 milliseconds using SolrJ client. This
> > application tries to delete the possible duplicates in each update
> window,
> > but we are using solr’s de-duplication techniques
> > <https://emea01.safelinks.protection.outlook.com/?url=
> > https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%
> > 2Fsolr%2FDe-Duplication&data=02%7C01%7Cdortega%40idealista.com%
> > 7Cb169ea024abc4954927208d4bc6868eb%7Cd78b7929c2a34897ae9a7d8f8dc1
> > a1cf%7C0%7C0%7C636340604697721266&sdata=WEhzoHC1Bf77K706%
> > 2Fj2wIWOw5gzfOgsP1IPQESvMsqQ%3D&reserved=0>
> >  too.
> >
> >
> >
> > We are indexing ~100 documents per second (with peaks of ~1000 documents
> > per second).
> >
> >
> >
> > Every search query is centralized in other application which exposes a
> DSL
>

Re: Excessive resources consumption migrating from Solr 6.6.0 Master/Slave to SolrCloud 6.6.0 (dozen times more resources)

[Daniel Ortega]

Hi Scott,

In our indexing service we are using that client too
(org.apache.solr.client.solrj.impl.CloudSolrClient) :)

This is out Update Request Processor chain configuration:

 true 
hash false solr.processor.Lookup3Signature  <
updateRequestProcessorChain processor="signature" name="dedupe"> <
requestHandler name="/update" class="solr.UpdateRequestHandler" >  dedupe  

Thanks for your reply :)

- Dani

2017-08-24 14:49 GMT+02:00 Scott Stults :

> Hi Daniel,
>
> SolrJ has a few client implementations to choose from: CloudSolrClient,
> ConcurrentUpdateSolrClient, HttpSolrClient, LBHttpSolrClient. You said your
> query service uses CloudSolrClient, but it would be good to verify which
> implementation your indexing service uses.
>
> One of the problems you might be having is with your deduplication step.
> Can you post your Update Request Processor Chain?
>
>
> -Scott
>
>
> On Wed, Aug 23, 2017 at 4:13 PM, Daniel Ortega <
> danielortegauf...@gmail.com>
> wrote:
>
> > Hi Scott,
> >
> > - *Can you describe the process that queries the DB and sends records to
> *
> > *Solr?*
> >
> > We are enqueueing ids during every ORACLE transaction (in
> insert/updates).
> >
> > An application dequeues every id and perform queries against dozen of
> > tables in the relational model to retrieve the fields to build the
> > document.  As we know that we are modifying the same ORACLE row in
> > different (but consecutive) transactions, we store only the last version
> of
> > the modified documents in a map data structure.
> >
> > The application has a configurable interval to send the documents stored
> in
> > the map to the update handler (we have tested different intervals from
> few
> > milliseconds to several seconds) using the SolrJ client. Actually we are
> > sending all the documents every 15 seconds.
> >
> > This application is developed using Java, Spring and Maven and we have
> > several instances.
> >
> > -* Is it a SolrJ-based application?*
> >
> > Yes, it is. We aren't using the last version of SolrJ client (we are
> > currently using SolrJ v6.3.0).
> >
> > - *If it is, which client package are you using?*
> >
> > I don't know exactly what do you mean saying 'client package' :)
> >
> > - *How many documents do you send at once?*
> >
> > It depends on the defined interval described before and the number of
> > transactions executed in our relational database. From dozens to few
> > hundreds (and even thousands).
> >
> > - *Are you sending your indexing or query traffic through a load
> balancer?*
> >
> > We aren't using a load balancer for indexing, but we have all our Rest
> > Query services through an HAProxy (using 'leastconn' algorithm). The Rest
> > Query Services performs queries using the CloudSolrClient.
> >
> > Thanks for your reply,
> > if you need any further information don't hesitate to ask
> >
> > Daniel
> >
> > 2017-08-23 14:57 GMT+02:00 Scott Stults  opensourceconnections.com
> > >:
> >
> > > Hi Daniel,
> > >
> > > Great background information about your setup! I've got just a few more
> > > questions:
> > >
> > > - Can you describe the process that queries the DB and sends records to
> > > Solr?
> > > - Is it a SolrJ-based application?
> > > - If it is, which client package are you using?
> > > - How many documents do you send at once?
> > > - Are you sending your indexing or query traffic through a load
> balancer?
> > >
> > > If you're sending documents to each replica as fast as they can take
> > them,
> > > you might be seeing a bottleneck at the shard leaders. The SolrJ
> > > CloudSolrClient finds out from Zookeeper which nodes are the shard
> > leaders
> > > and sends docs directly to them.
> > >
> > >
> > > -Scott
> > >
> > > On Tue, Aug 22, 2017 at 2:16 PM, Daniel Ortega <
> > > danielortegauf...@gmail.com>
> > > wrote:
> > >
> > > > *Main Problems*
> > > >
> > > >
> > > > We are involved in a migration from Solr Master/Slave infrastructure
> to
> > > > SolrCloud infrastructure.
> > > >
> > > >
> > > >
> > > > The main problems that we have now are:
> > > >
> > > >
> > > >
> > > >- Excessive resources consumption:

Re: Excessive resources consumption migrating from Solr 6.6.0 Master/Slave to SolrCloud 6.6.0 (dozen times more resources)

[Daniel Ortega]

Hi Scott,

Yes, we think that our usage scenario falls into Index-Heavy/Query-Heavy
too. We have tested with several values in softcommit/hardcommit values
(from few seconds to minutes) with no appreciable improvements :(

Thanks for your reply!

- Daniel

2017-08-25 6:45 GMT+02:00 Scott Stults :

> Hi Dani,
>
> It seems like your use case falls into the Index-Heavy / Query-Heavy
> category, so you might try increasing your hard commit frequency to 15
> seconds rather than 15 minutes:
>
> https://lucidworks.com/2013/08/23/understanding-
> transaction-logs-softcommit-and-commit-in-sorlcloud/
>
>
> -Scott
>
> On Thu, Aug 24, 2017 at 10:03 AM, Daniel Ortega <
> danielortegauf...@gmail.com
> > wrote:
>
> > Hi Scott,
> >
> > In our indexing service we are using that client too
> > (org.apache.solr.client.solrj.impl.CloudSolrClient) :)
> >
> > This is out Update Request Processor chain configuration:
> >
> >  > name
> > ="signature"> true  name="signatureField">
> > hash false  > "signatureClass">solr.processor.Lookup3Signature
> 
> > <
> > updateRequestProcessorChain processor="signature" name="dedupe">
>  > class="solr.LogUpdateProcessorFactory" />  > "solr.RunUpdateProcessorFactory" />   <
> > requestHandler name="/update" class="solr.UpdateRequestHandler" >  > name=
> > "defaults"> dedupe 
> 
> >
> > Thanks for your reply :)
> >
> > - Dani
> >
> > 2017-08-24 14:49 GMT+02:00 Scott Stults  opensourceconnections.com
> > >:
> >
> > > Hi Daniel,
> > >
> > > SolrJ has a few client implementations to choose from: CloudSolrClient,
> > > ConcurrentUpdateSolrClient, HttpSolrClient, LBHttpSolrClient. You said
> > your
> > > query service uses CloudSolrClient, but it would be good to verify
> which
> > > implementation your indexing service uses.
> > >
> > > One of the problems you might be having is with your deduplication
> step.
> > > Can you post your Update Request Processor Chain?
> > >
> > >
> > > -Scott
> > >
> > >
> > > On Wed, Aug 23, 2017 at 4:13 PM, Daniel Ortega <
> > > danielortegauf...@gmail.com>
> > > wrote:
> > >
> > > > Hi Scott,
> > > >
> > > > - *Can you describe the process that queries the DB and sends records
> > to
> > > *
> > > > *Solr?*
> > > >
> > > > We are enqueueing ids during every ORACLE transaction (in
> > > insert/updates).
> > > >
> > > > An application dequeues every id and perform queries against dozen of
> > > > tables in the relational model to retrieve the fields to build the
> > > > document.  As we know that we are modifying the same ORACLE row in
> > > > different (but consecutive) transactions, we store only the last
> > version
> > > of
> > > > the modified documents in a map data structure.
> > > >
> > > > The application has a configurable interval to send the documents
> > stored
> > > in
> > > > the map to the update handler (we have tested different intervals
> from
> > > few
> > > > milliseconds to several seconds) using the SolrJ client. Actually we
> > are
> > > > sending all the documents every 15 seconds.
> > > >
> > > > This application is developed using Java, Spring and Maven and we
> have
> > > > several instances.
> > > >
> > > > -* Is it a SolrJ-based application?*
> > > >
> > > > Yes, it is. We aren't using the last version of SolrJ client (we are
> > > > currently using SolrJ v6.3.0).
> > > >
> > > > - *If it is, which client package are you using?*
> > > >
> > > > I don't know exactly what do you mean saying 'client package' :)
> > > >
> > > > - *How many documents do you send at once?*
> > > >
> > > > It depends on the defined interval described before and the number of
> > > > transactions executed in our relational database. From dozens to few
> > > > hundreds (and even thousands).
> > > >
> > > > - *Are you sending your indexing or query traffic through a load
> > > balancer?*
> > > >
> > > > We aren't using a load balancer for indexing, but we have all our
> Rest
&

Re: Performance Test

[Daniel Ortega]

I would recommend you Solrmeter cloud

This fork supports solr cloud:
https://github.com/idealista/solrmeter/blob/master/README.md

Disclaimer: This fork was developed by idealista, the company where I work

El El lun, 4 sept 2017 a las 11:18, Selvam Raman 
escribió:

> Hi All,
>
> which is the best tool for solr perfomance test. I want to identify how
> much load my solr could handle and how many concurrent users can query on
> solr.
>
> Please suggest.
>
> --
> Selvam Raman
> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>

Re: Anyone have any comments on current solr monitoring favorites?

[Daniel Ortega]

Hi Robert,

We use the following stack:

- Prometheus to scrape metrics (https://prometheus.io/)
- Prometheus node exporter to export "machine metrics" (Disk, network
usage, etc.) (https://github.com/prometheus/node_exporter)
- Prometheus JMX exporter to export "Solr metrics" (Cache usage, QPS,
Response times...) (https://github.com/prometheus/jmx_exporter)
- Grafana to visualize all the data scrapped by Prometheus (
https://grafana.com/)

Best regards
Daniel Ortega

2017-11-06 20:13 GMT+01:00 Petersen, Robert (Contr) <
robert.peters...@ftr.com>:

> PS I knew sematext would be required to chime in here!  😊
>
>
> Is there a non-expiring dev version I could experiment with? I think I did
> sign up for a trial years ago from a different company... I was actually
> wondering about hooking it up to my personal AWS based solr cloud instance.
>
>
> Thanks
>
> Robi
>
> 
> From: Emir Arnautović 
> Sent: Thursday, November 2, 2017 2:05:10 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Anyone have any comments on current solr monitoring favorites?
>
> Hi Robi,
> Did you try Sematext’s SPM? It provides host, JVM and Solr metrics and
> more. We use it for monitoring our Solr instances and for consulting.
>
> Disclaimer - see signature :)
>
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
> > On 2 Nov 2017, at 19:35, Walter Underwood  wrote:
> >
> > We use New Relic for JVM, CPU, and disk monitoring.
> >
> > I tried the built-in metrics support in 6.4, but it just didn’t do what
> we want. We want rates and percentiles for each request handler. That gives
> us 95th percentile for textbooks suggest or for homework search results
> page, etc. The Solr metrics didn’t do that. The Jetty metrics didn’t do
> that.
> >
> > We built a dedicated servlet filter that goes in front of the Solr
> webapp and reports metrics. It has some special hacks to handle some weird
> behavior in SolrJ. A request to the “/srp” handler is sent as
> “/select?qt=/srp”, so we normalize that.
> >
> > The metrics start with the cluster name, the hostname, and the
> collection. The rest is generated like this:
> >
> > URL: GET /solr/textbooks/select?q=foo&qt=/auto
> > Metric: textbooks.GET./auto
> >
> > URL: GET /solr/textbooks/select?q=foo
> > Metric: textbooks.GET./select
> >
> > URL: GET /solr/questions/auto
> > Metric: questions.GET./auto
> >
> > So a full metric for the cluster “solr-cloud” and the host “search01"
> would look like “solr-cloud.search01.solr.textbooks.GET./auto.m1_rate”.
> >
> > We send all that to InfluxDB. We’ve configured a template so that each
> part of the metric name is mapped to a field, so we can write efficient
> queries in InfluxQL.
> >
> > Metrics are graphed in Grafana. We have dashboards that mix Cloudwatch
> (for the load balancer) and InfluxDB.
> >
> > I’m still working out the kinks in some of the more complicated queries,
> but the data is all there. I also want to expand the servlet filter to
> report HTTP response codes.
> >
> > wunder
> > Walter Underwood
> > wun...@wunderwood.org
> > http://observer.wunderwood.org/  (my blog)
> >
> >
> >> On Nov 2, 2017, at 9:30 AM, Petersen, Robert (Contr) <
> robert.peters...@ftr.com> wrote:
> >>
> >> OK I'm probably going to open a can of worms here...  lol
> >>
> >>
> >> In the old old days I used PSI probe to monitor solr running on tomcat
> which worked ok on a machine by machine basis.
> >>
> >>
> >> Later I had a grafana dashboard on top of graphite monitoring which was
> really nice looking but kind of complicated to set up.
> >>
> >>
> >> Even later I successfully just dropped in a newrelic java agent which
> had solr monitors and a dashboard right out of the box, but it costs money
> for the full tamale.
> >>
> >>
> >> For basic JVM health and Solr QPS and time percentiles, does anyone
> have any favorites or other alternative suggestions?
> >>
> >>
> >> Thanks in advance!
> >>
> >> Robi
> >>
> >> 
> >>
> >> This communication is confidential. Frontier only sends and receives
> email on the basis of the terms set out at http://www.frontier.com/email_
> disclaimer.
> >
>
>

Re: Anyone have any comments on current solr monitoring favorites?

[Daniel Ortega]

Hi @Atita,

We are using the latear version (Solr 7.1.0).

As the metrics are exposed with MBeans via JMX, you could use the
Prometheus JMX exportar to take the values of that metrics and expose them.
You could use it to monitor caches, response times, number of errors in all
the handlers you have defined.

To configure JMX in a Solr instance follow this link:
https://lucene.apache.org/solr/guide/6_6/using-jmx-with-solr.html

This page explains some of the JMX metrics that Solr exposes:
https://lucene.apache.org/solr/guide/6_6/performance-statistics-reference.html

Basically the JMX exporter is an Embedded Jetty server that read values
exposed using JMX (in localhost or in a remote instance), parse that values
and exposes them using the format that Prometheus could scrap.

Best regards,
Daniel

El El mar, 7 nov 2017 a las 2:43, Atita Arora 
escribió:

> Hi @Daniel ,
>
> What version of Solr are you using ?
> We gave prometheus + Jolokia + InfluxDB + Grafana a try , that came out
> well.
> With Solr 6.6 the metrics are explosed through the /metrics api, but how do
> we go about for the earlier versions , please guide ?
> Specifically the cache monitoring.
>
> Thanks in advance,
> Atita
>
> On Mon, Nov 6, 2017 at 2:19 PM, Daniel Ortega  >
> wrote:
>
> > Hi Robert,
> >
> > We use the following stack:
> >
> > - Prometheus to scrape metrics (https://prometheus.io/)
> > - Prometheus node exporter to export "machine metrics" (Disk, network
> > usage, etc.) (https://github.com/prometheus/node_exporter)
> > - Prometheus JMX exporter to export "Solr metrics" (Cache usage, QPS,
> > Response times...) (https://github.com/prometheus/jmx_exporter)
> > - Grafana to visualize all the data scrapped by Prometheus (
> > https://grafana.com/)
> >
> > Best regards
> > Daniel Ortega
> >
> > 2017-11-06 20:13 GMT+01:00 Petersen, Robert (Contr) <
> > robert.peters...@ftr.com>:
> >
> > > PS I knew sematext would be required to chime in here!  😊
> > >
> > >
> > > Is there a non-expiring dev version I could experiment with? I think I
> > did
> > > sign up for a trial years ago from a different company... I was
> actually
> > > wondering about hooking it up to my personal AWS based solr cloud
> > instance.
> > >
> > >
> > > Thanks
> > >
> > > Robi
> > >
> > > 
> > > From: Emir Arnautović 
> > > Sent: Thursday, November 2, 2017 2:05:10 PM
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: Anyone have any comments on current solr monitoring
> > favorites?
> > >
> > > Hi Robi,
> > > Did you try Sematext’s SPM? It provides host, JVM and Solr metrics and
> > > more. We use it for monitoring our Solr instances and for consulting.
> > >
> > > Disclaimer - see signature :)
> > >
> > > Emir
> > > --
> > > Monitoring - Log Management - Alerting - Anomaly Detection
> > > Solr & Elasticsearch Consulting Support Training -
> http://sematext.com/
> > >
> > >
> > >
> > > > On 2 Nov 2017, at 19:35, Walter Underwood 
> > wrote:
> > > >
> > > > We use New Relic for JVM, CPU, and disk monitoring.
> > > >
> > > > I tried the built-in metrics support in 6.4, but it just didn’t do
> what
> > > we want. We want rates and percentiles for each request handler. That
> > gives
> > > us 95th percentile for textbooks suggest or for homework search results
> > > page, etc. The Solr metrics didn’t do that. The Jetty metrics didn’t do
> > > that.
> > > >
> > > > We built a dedicated servlet filter that goes in front of the Solr
> > > webapp and reports metrics. It has some special hacks to handle some
> > weird
> > > behavior in SolrJ. A request to the “/srp” handler is sent as
> > > “/select?qt=/srp”, so we normalize that.
> > > >
> > > > The metrics start with the cluster name, the hostname, and the
> > > collection. The rest is generated like this:
> > > >
> > > > URL: GET /solr/textbooks/select?q=foo&qt=/auto
> > > > Metric: textbooks.GET./auto
> > > >
> > > > URL: GET /solr/textbooks/select?q=foo
> > > > Metric: textbooks.GET./select
> > > >
> > > > URL: GET /solr/questions/auto
> > > > Metric: questions.GET./auto
> > > >
> > > > So a full metric for the cluster “solr-cloud” and the host “search01"
> &