This is an automated email from the ASF dual-hosted git repository.
ptoth pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new cdd2790 [SPARK-53187] Support SparkCluster event related metrics set
cdd2790 is described below
commit cdd279006b95e23a24810ebe0c064b4f06363617
Author: Zhou JIANG <jiang...@umich.edu>
AuthorDate: Fri Aug 8 13:13:33 2025 +0200
[SPARK-53187] Support SparkCluster event related metrics set
### What changes were proposed in this pull request?
This PR adds SparkCluster as a supported type in metrics and therefore
supports publishing execution metrics for it.
### Why are the changes needed?
Operator by default publish metrics by resource type (SparkApplication) -
by adding this support, we support the same set of counter and histogram for
received event and timed execution
### Does this PR introduce _any_ user-facing change?
More Metrics becomes available for SparkClusters
### How was this patch tested?
From dev sandbox we can see metrics like
```
metrics_operator.sdk_sparkapplication_added_resource_event_Count{type="counters"}
9010
metrics_operator.sdk_sparkapplication_reconciliation_failed_Count{type="counters"}
9
metrics_operator.sdk_sparkapplication_reconciliation_finished_Count{type="counters"}
182841
metrics_operator.sdk_sparkapplication_reconciliation_retries_Count{type="counters"}
9
metrics_operator.sdk_sparkcluster_added_resource_event_Count{type="counters"}
9009
metrics_operator.sdk_sparkcluster_reconciliation_failed_Count{type="counters"} 0
metrics_operator.sdk_sparkcluster_reconciliation_finished_Count{type="counters"}
182821
metrics_operator.sdk_sparkcluster_reconciliation_retries_Count{type="counters"} 0
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #295 from jiangzho/cluster_metrics.
Authored-by: Zhou JIANG <jiang...@umich.edu>
Signed-off-by: Peter Toth <p_t...@apple.com>
---
.../apache/spark/k8s/operator/metrics/source/OperatorJosdkMetrics.java | 3 +++
1 file changed, 3 insertions(+)
diff --git
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/metrics/source/OperatorJosdkMetrics.java
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/metrics/source/OperatorJosdkMetrics.java
index 86a34ff..eee97a4 100644
---
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/metrics/source/OperatorJosdkMetrics.java
+++
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/metrics/source/OperatorJosdkMetrics.java
@@ -44,6 +44,7 @@ import lombok.extern.slf4j.Slf4j;
import org.apache.spark.k8s.operator.BaseResource;
import org.apache.spark.k8s.operator.SparkApplication;
+import org.apache.spark.k8s.operator.SparkCluster;
import org.apache.spark.metrics.source.Source;
import org.apache.spark.util.Clock;
import org.apache.spark.util.SystemClock;
@@ -305,6 +306,8 @@ public class OperatorJosdkMetrics implements Source,
Metrics {
if (resourceGvk.getKind().equals(SparkApplication.class.getSimpleName())) {
resourceClass = SparkApplication.class;
+ } else if
(resourceGvk.getKind().equals(SparkCluster.class.getSimpleName())) {
+ resourceClass = SparkCluster.class;
} else {
return Optional.empty();
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
