It's the same as calculating the total incoming request rate of N pods in a
deployment: sum(rate(grpc_request_count{service=foo}[5m]))For our use case, we have N kubernetes jobs being spun up pushing many metrics, like grpc_request_count, to pushgateway. If I want to know the rate of outgoing requests, I have to have unique labels on the metric for prometheus to accurately calculate the request rate of the service as a whole. Without them they just overwrite each other. For pushgateway, though, those unique ids are never GCd like they are in prometheus. So, every day a service creating 2 pods every 5 minutes and writing just a single grpc_request_count metric with 20 label values to pushgateway will have a cardinality of (2 pod ids * 288 times a day * 20 grpc_request_count labels). That's a cardinality of 11k series per day for that one metric from that one service that will not be GC'd by pushgateway until I restart it. On Thu, Jun 29, 2023 at 7:03 AM Bjoern Rabenstein <[email protected]> wrote: > On 14.06.23 13:10, 'Braden Schaeffer' via Prometheus Developers wrote: > > > > The most basic example, two batch jobs that produce the same metrics > (grpc > > or http metrics). This is not just `last_completed_at` or something as I > > have seen before where its the same metric being updated over and over > > agin. You have to include a label that identifies these jobs as > different > > so that metrics like gRPC request rates can be calculated correctly. In > the > > kubernetes world this usually means pod ID. Simple enough until you have > > 1000s of these pod IDs compounded by other labels. > > I don't fully understand what you are trying to do. Could you explain > what metrics you are pushing exactly, and what PromQL expressions you > are using to "correctly calculate a gRPC request rate"? > > -- > Björn Rabenstein > [PGP-ID] 0x851C3DA17D748D03 > [email] [email protected] > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAKG2A1cgwqxMQxVYKkBK6r3crFDOh3iZckCbMhuYUBSbLwwVwg%40mail.gmail.com.
