With respect to OpenMetrics push, we had something very similar at $prevco that pushed something that looked very similar to the protobuf payload of OpenMetrics (but was Thrift snapshot of an aggregated set of metrics from in process) that was used by short running tasks (for Jenkins, Flink jobs, etc).
I definitely agree it’s not ideal and ideally the platform provider can supply a collection point (there is something for Jenkins, a plug-in that can do this, but custom metrics is very hard / nigh impossible to make work with it, and this is a non-cloud provider environment that’s actually possible to make work, just no one has made it seamless). I agree with Richi that something that could push to a Prometheus Agent like target that supports OpenMetrics push could be a good middle ground with the right support / guidelines: - A way to specify multiple Prometheus Agent targets and quickly failover from one to another if within $X ms one is not responding (you could imagine a 5ms budget for each and max 3 are tried, introducing at worst 15ms overhead when all are down in 3 local availability zones, but in general this is a disaster case) - Deduplication ability so that a retried push is not double counted, this might mean timestamping the metrics… (so if written twice only first record kept, etc) I think it should similar to the Push Gateway be generally a last resort kind of option and have clear limitations so that pull still remains the clear choice for anything but these environments. Is there any interest discussing this on a call some time? Rob On Thu, Jun 24, 2021 at 5:09 PM Bjoern Rabenstein <[email protected]> wrote: > On 22.06.21 11:26, Tobias Schmidt wrote: > > > > Last night I was wondering if there are any other common interfaces > > available in serverless environments and noticed that all products by AWS > > (Lambda) and GCP (Functions, Run) at least provide the option to handle > log > > streams, sometimes even log files on disk. I'm currently thinking about > > experimenting with an approach where containers log metrics to stdout / > > some file, get picked up by the serverless runtime and written to some > log > > stream. Another service "loggateway" (or otherwise named) would then > stream > > the logs, aggregate them and either expose them on the common /metrics > > endpoint or push them with remote write right away to a Prometheus > instance > > hosted somewhere (like Grafana Cloud). > > Perhaps I'm missing something, but isn't that > https://github.com/google/mtail ? > > -- > Björn Rabenstein > [PGP-ID] 0x851C3DA17D748D03 > [email] [email protected] > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/20210624210908.GB11559%40jahnn > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CABakzZaVwUW27uSvto%2BVkPtspcKP2B4pCkSHQi-%2B1C%2Bb22R1yg%40mail.gmail.com.
