By default, Solr returns all metrics, of which there are a ton of them!
That's great for looking at metrics if you are exploring them for the first
time. But as you'll see in the ref guide, there are many filtering options
which I would consider to be mandatory if you are going to use metrics "for
real" (in production).
I know I sidestepped your question; you may have a point about indentation
defaults. I'm somewhat surprised the Prometheus Exporter insists on JSON
(vs Solr native "javabin"); I can see the code for that and it seems it's
because it's going to do some Json specific "jq" processing. Okay.
See org.apache.solr.prometheus.scraper.SolrScraper#request where you could
clone the params from MetricsQuery in order to set indent=false. Or be a
little fancy and use SolrParams.wrapDefaults(params, new
MapSolrParams(Map.of("indent","false"))) or similar. I wish there was a
convenience method for that on SolrParams.
Feel free to create a JIRA issue & PR and I'll review.
~ David
On Thu, Sep 28, 2023 at 1:38 PM Chris Schomaker
<chris.schoma...@reddit.com.invalid> wrote:
> I'd like to raise an issue that I'm having with Solr 9.3 that might make a
> good case for getting another release (9.3.1 or 9.4) out soon.
>
> I've had a 9.3 cluster of 180 nodes up and running for about a week. After
> the weekend, I noticed that metrics from the cluster were failing to ship.
> Upon investigation, I found that the payload coming back from
> `/solr/admin/metrics` was over a GB in size. I also saw query responses had
> ballooned in size to tens of MB and the UI was slow. I captured query
> responses in the browser, with curl and in a packet capture. In all cases,
> there were a large amount of space characters. This can be seen in the
> brower's network inspector when clicking around the UI, as well (make sure
> to disable pretty print).
>
> This all lead me to SOLR-16925. I have a fix in place for query results but
> the admin endpoints like `/solr/admin/metrics` are still experiencing the
> issue. You can simply pass `&indent=off` to the endpoints and the
> whitespace issue goes away. However, I can't figure out a way to configure
> Solr to use this default param with the CoreAdminHandler. The Solr Prom
> Exporter could be reconfigured to include this param but that's a lot of
> configuration to add. There's no way to reconfigure the UI.
>
> Here's a good illustration of the issue:
>
> ```
> ❯ curl --user 'foo:bar' "http://localhost:8983/solr/admin/metrics"; >
> /tmp/metrics_indented.json
>
> ❯ curl --user 'foo:bar' "
> http://localhost:8983/solr/admin/metrics?indent=off";
> > /tmp/metrics_not_indented.json
>
> ❯ ls -lh /tmp | grep metrics
> -rw-rw-r-- 1 foo bar 1.4G Sep 28 08:57 metrics_indented.json
> -rw-rw-r-- 1 foo bar 177K Sep 28 08:59 metrics_not_indented.json
>
> ❯ grep -o ' ' /tmp/metrics_indented.json | wc -l
> 1449499825
> ```
>
> In my opinion, Solr 9.3 can't be monitored and is therefore unfit to run.
>
> I'll be AFK for the next few days but am happy to lend a hand with a new
> release next week. I'd need a lot of help, though.
>
> Thanks,
> Chris
>
