This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new d8d7bbe125 Add some catch-all rules (#11539)
d8d7bbe125 is described below
commit d8d7bbe12573514df171e09c90880d5ac00616b4
Author: Gonzalo Ortiz Jaureguizar <[email protected]>
AuthorDate: Sun Sep 10 13:21:59 2023 +0200
Add some catch-all rules (#11539)
---
.../pinot/etc/jmx_prometheus_javaagent/README.md | 16 +++++++++--
.../jmx_prometheus_javaagent/configs/broker.yml | 28 ++++++++++++++++--
.../configs/controller.yml | 28 ++++++++++++++++--
.../jmx_prometheus_javaagent/configs/minion.yml | 27 ++++++++++++++++++
.../etc/jmx_prometheus_javaagent/configs/pinot.yml | 33 ++++++++++++++++++++++
.../jmx_prometheus_javaagent/configs/server.yml | 28 ++++++++++++++++--
6 files changed, 148 insertions(+), 12 deletions(-)
diff --git a/docker/images/pinot/etc/jmx_prometheus_javaagent/README.md
b/docker/images/pinot/etc/jmx_prometheus_javaagent/README.md
index bcb6b34801..e1d1f059e1 100644
--- a/docker/images/pinot/etc/jmx_prometheus_javaagent/README.md
+++ b/docker/images/pinot/etc/jmx_prometheus_javaagent/README.md
@@ -6,17 +6,27 @@ Prometheus is not a full-fledged dashboarding solution and
needs to be hooked up
#### How do Pinot metrics end up in Prometheus?
-Currently, Pinot metrics are exposed as JMX mbeans through the
PinotJmxReporter. These JMX mbeans are consumed by Prometheus using the
[Prometheus JMX Exporter](https://github.com/prometheus/jmx_exporter). A fairly
comprehensive Prometheus JMX Exporter config can be found under
`docker/images/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml`.
+Currently, Pinot metrics are exposed as JMX mbeans through the
PinotJmxReporter.
+These JMX mbeans are consumed by Prometheus using the [Prometheus JMX
Exporter](https://github.com/prometheus/jmx_exporter).
+Fairly comprehensive Prometheus JMX Exporter config files can be found under
+[docker/images/pinot/etc/jmx_prometheus_javaagent/configs/](docker/images/pinot/etc/jmx_prometheus_javaagent/configs/).
See the [Pinot
docs](https://docs.pinot.apache.org/operators/operating-pinot/monitoring) for
more info.
#### How can I view and test metrics?
-First, you need to make sure the metrics are exposed though JMX. Note, that if
no metric has been published (e.g. no values recorded), the metric will not
show up in JMX or Prometheus server.
-With a local Pinot deployment, you can launch `jconsole`, select your local
deployment and view all the metrics exposed as jmx mbeans. To see if the
metrics are being consumed with the Prometheus JMX Exporter and your config
file, you can set the JAVA_OPTS env variable before running Pinot locally.
+First, you need to make sure the metrics are exposed though JMX.
+Note, that if no metric has been published (e.g. no values recorded), the
metric will not show up in JMX or Prometheus server.
+With a local Pinot deployment, you can launch `jconsole`, select your local
deployment and view all the metrics exposed as jmx mbeans.
+Alternative, you can use [jmxterm](https://docs.cyclopsgroup.org/jmxterm) in
order to read them using a CLI.
+To see if the metrics are being consumed with the Prometheus JMX Exporter and
your config file,
+you can set the JAVA_OPTS env variable before running Pinot locally.
`export JAVA_OPTS="-javaagent:jmx_prometheus_javaagent.jar=8080:pinot.yml
-Xms4G -Xmx4G -XX:MaxDirectMemorySize=30g
-Dlog4j2.configurationFile=conf/pinot-admin-log4j2.xml
-Dplugins.dir=$BASEDIR/plugins"
bin/pinot-admin.sh ....
`
+Remember that it is recommended to use service specific prometheus
configurations (like `broker.yml`, `server.yml`, etc)
+in production instead of `pinot.xml`.
+
This will expose a port at 8080 to dump metrics as Prometheus format for
Prometheus scraper to fetch.
diff --git
a/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/broker.yml
b/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/broker.yml
index 91e72bc39b..543dabcd15 100644
--- a/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/broker.yml
+++ b/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/broker.yml
@@ -144,8 +144,30 @@ rules:
cache: true
labels:
table: "$1"
-- pattern: "\"org.apache.pinot.common.metrics\"<type=\"BrokerMetrics\",
name=\"pinot.broker.version.(\\w+)\"><>(\\w+)"
- name: "pinot_broker_version"
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.version\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_version"
cache: true
labels:
- version: "$1"
+ version: "$2"
+
+ ## Metrics that fit the catch-all patterns above should not be added to this
file.
+ ## In case a metric does not fit the catch-all patterns, add them before
this comment
+
+ # This is a catch-all pattern for pinot table metrics with offline/realtime
suffix.
+ # Patterns after this line may be skipped.
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.(\\w+)_(OFFLINE|REALTIME)\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_$4_$5"
+ cache: true
+ labels:
+ table: "$2"
+ tableType: "$3"
+ # This is a catch-all pattern for pinot table metrics. Patterns after this
line may be skipped.
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.(\\w+)\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_$3_$4"
+ cache: true
+ labels:
+ table: "$2"
+ # This is a catch-all pattern for pinot controller metrics not related to
tables. Patterns after this line may be skipped.
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_$2_$3"
+ cache: true
diff --git
a/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/controller.yml
b/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/controller.yml
index 88ea8dff92..cfd4f9af31 100644
--- a/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/controller.yml
+++ b/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/controller.yml
@@ -193,8 +193,30 @@ rules:
labels:
table: "$1"
tableType: "$2"
-- pattern: "\"org.apache.pinot.common.metrics\"<type=\"ControllerMetrics\",
name=\"pinot.controller.version.(\\w+)\"><>(\\w+)"
- name: "pinot_controller_version"
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.version\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_version"
cache: true
labels:
- version: "$1"
+ version: "$2"
+
+ ## Metrics that fit the catch-all patterns above should not be added to this
file.
+ ## In case a metric does not fit the catch-all patterns, add them before
this comment
+
+ # This is a catch-all pattern for pinot table metrics with offline/realtime
suffix.
+ # Patterns after this line may be skipped.
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.(\\w+)_(OFFLINE|REALTIME)\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_$4_$5"
+ cache: true
+ labels:
+ table: "$2"
+ tableType: "$3"
+ # This is a catch-all pattern for pinot table metrics. Patterns after this
line may be skipped.
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.(\\w+)\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_$3_$4"
+ cache: true
+ labels:
+ table: "$2"
+ # This is a catch-all pattern for pinot controller metrics not related to
tables. Patterns after this line may be skipped.
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_$2_$3"
+ cache: true
diff --git
a/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/minion.yml
b/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/minion.yml
index 95853f50ec..584cde5963 100644
--- a/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/minion.yml
+++ b/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/minion.yml
@@ -25,3 +25,30 @@ rules:
cache: true
labels:
id: "$1"
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.version\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_version"
+ cache: true
+ labels:
+ version: "$2"
+
+ ## Metrics that fit the catch-all patterns above should not be added to this
file.
+ ## In case a metric does not fit the catch-all patterns, add them before
this comment
+
+ # This is a catch-all pattern for pinot table metrics with offline/realtime
suffix.
+ # Patterns after this line may be skipped.
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.(\\w+)_(OFFLINE|REALTIME)\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_$4_$5"
+ cache: true
+ labels:
+ table: "$2"
+ tableType: "$3"
+ # This is a catch-all pattern for pinot table metrics. Patterns after this
line may be skipped.
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.(\\w+)\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_$3_$4"
+ cache: true
+ labels:
+ table: "$2"
+ # This is a catch-all pattern for pinot controller metrics not related to
tables. Patterns after this line may be skipped.
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_$2_$3"
+ cache: true
diff --git a/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml
b/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml
index a502742e02..6660ee42d7 100644
--- a/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml
+++ b/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml
@@ -1,3 +1,8 @@
+# It is recommended to use the specific service yml file (like broker.yml,
controller.yml, etc)
+# instead of this file.
+# Prometheus executes rules in a linear time, so the more rules your
configuration has, the more CPU will be spent by
+# Prometheus Java agent.
+
rules:
# Pinot Controller
- pattern: "\"org.apache.pinot.common.metrics\"<type=\"ControllerMetrics\",
name=\"pinot.controller.controller(\\w+)\"><>(\\w+)"
@@ -482,3 +487,31 @@ rules:
cache: true
labels:
id: "$1"
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.version\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_version"
+ cache: true
+ labels:
+ version: "$2"
+
+
+ ## Metrics that fit the catch-all patterns above should not be added to this
file.
+ ## In case a metric does not fit the catch-all patterns, add them before
this comment
+
+ # This is a catch-all pattern for pinot table metrics with offline/realtime
suffix.
+ # Patterns after this line may be skipped.
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.(\\w+)_(OFFLINE|REALTIME)\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_$4_$5"
+ cache: true
+ labels:
+ table: "$2"
+ tableType: "$3"
+ # This is a catch-all pattern for pinot table metrics. Patterns after this
line may be skipped.
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.(\\w+)\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_$3_$4"
+ cache: true
+ labels:
+ table: "$2"
+ # This is a catch-all pattern for pinot controller metrics not related to
tables. Patterns after this line may be skipped.
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_$2_$3"
+ cache: true
diff --git
a/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/server.yml
b/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/server.yml
index ad5b96eef8..d02345dc49 100644
--- a/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/server.yml
+++ b/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/server.yml
@@ -125,8 +125,30 @@ rules:
table: "$1"
tableType: "$2"
partition: "$3"
-- pattern: "\"org.apache.pinot.common.metrics\"<type=\"ServerMetrics\",
name=\"pinot.server.version.(\\w+)\"><>(\\w+)"
- name: "pinot_server_version"
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.version\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_version"
cache: true
labels:
- version: "$1"
+ version: "$2"
+
+ ## Metrics that fit the catch-all patterns above should not be added to this
file.
+ ## In case a metric does not fit the catch-all patterns, add them before
this comment
+
+ # This is a catch-all pattern for pinot table metrics with offline/realtime
suffix.
+ # Patterns after this line may be skipped.
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.(\\w+)_(OFFLINE|REALTIME)\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_$4_$5"
+ cache: true
+ labels:
+ table: "$2"
+ tableType: "$3"
+ # This is a catch-all pattern for pinot table metrics. Patterns after this
line may be skipped.
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.(\\w+)\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_$3_$4"
+ cache: true
+ labels:
+ table: "$2"
+ # This is a catch-all pattern for pinot controller metrics not related to
tables. Patterns after this line may be skipped.
+- pattern: "\"?org\\.apache\\.pinot\\.common\\.metrics\"?<type=\"?\\w+\"?,
name=\"?pinot\\.(\\w+)\\.(\\w+)\"?><>(\\w+)"
+ name: "pinot_$1_$2_$3"
+ cache: true
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
