Hi Bartek, Thanks for bringing this up! This is great news, infact we were going to look into the same in Cortex as well! For testing the new blocks based storage we were going to mirror writes from a subset of our users to the new storage engine and run both in parallel. We already have a query-tee <https://github.com/cortexproject/cortex/tree/master/cmd/query-tee> tool which mirrors queries, next is mirroring the write path for us. While we don't need the obfuscation let us all collaborate to create a proxy/mirror. And thanks for sharing the vmagent Aliaksandr, will definitely look into it.
Thanks, Goutham. On Monday, April 20, 2020 at 8:50:31 PM UTC+1, Aliaksandr Valialkin wrote: > > > just I am a bit concerned about one binary that enables everything. It > might be quite hard work to maintain it. > > vmagent is just easy to use metrics proxy, which performs the following > tasks: > - Accepts data via various popular ingestion protocols (Prometheus > remote_write, Influx line protocol, Graphite, OpenTSDB, CSV). Additionally, > it can scrape Prometheus targets. > - Augments and filters the accepted data with Prometheus-compatible > relabeling. > - Pushes the filtered data to the configured remote storage targets via > Prometheus remote_write protocol. Additional per-target relabeling can be > applied to data before sending it to each remote storage target. > > vmagent uses independent file-base buffers for each configured remote > storage target, so it may buffer the data locally until temporarily > unavailable remote storage target becomes available again. > > > On Mon, Apr 20, 2020 at 9:44 PM Bartłomiej Płotka <[email protected] > <javascript:>> wrote: > >> Thanks, Aliaksandr! >> >> So vmagent on top of scraping ALSO receive remote write API? 😱 What it >> CANNOT do? =D >> >> It looks indeed that feature-wise it is what we meant, just I am a bit >> concerned about one binary that enables everything. It might be quite hard >> work to maintain it... You must be some kind of superhuman Aliaksandr! (: >> Definitely will take a look, thanks. (: >> >> Kind Regards, >> Bartek >> >> On Mon, 20 Apr 2020 at 19:38, Aliaksandr Valialkin <[email protected] >> <javascript:>> wrote: >> >>> Such a mirroring can be done with vmagent >>> <https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md> >>> >>> - just configure multiple `-remoteWrite.url` targets with distinct >>> `-remoteWrite.urlRelabelConfig` configs for obfuscation and filtering. The >>> final system will look like the following: >>> >>> -> remote target1 (prod) >>> Prometheus -> vmagent -> filtering -> remote target2 (staging) >>> -> obfuscation -> remote target3 (dev) >>> >>> >>> On Mon, Apr 20, 2020 at 9:24 PM Bartłomiej Płotka <[email protected] >>> <javascript:>> wrote: >>> >>>> Hi! >>>> >>>> This question is not strictly related to Prometheus, but rather to >>>> server-side Remote Write APIs. >>>> We are looking at how to have more realistic staging environments for >>>> servers like that. In order to achieve so, we want to "mirror" / "fork" >>>> portion of production remote write traffic to other clusters APIs (e.g >>>> staging or dev environment). >>>> >>>> As part of this mirroring, data has to be potentially obfuscated to >>>> avoid leaking of sensitive data, but also without totally changing the >>>> characteristic of data (e.g same number labels, labels values/names with >>>> the same amount of characters, etc). >>>> >>>> In the future, we could add some more advanced features if needed (e.g >>>> load balancing). >>>> >>>> Wonder if anyone in the community had been working on something like >>>> that already and has something to share/is already shared? >>>> >>>> ProxySQL <https://github.com/sysown/proxysql/wiki/Mirroring> is >>>> something like that but in the SQL world. Would be awesome to have the >>>> same >>>> for remote write (and Query API as well I guess, but let's think about it >>>> in a separate thread) (: >>>> <https://github.com/thanos-io/thanos/issues/2480> >>>> Some discussion on Thanos project: >>>> https://github.com/thanos-io/thanos/issues/2480 >>>> >>>> Please help if you know or have worked on something like this (: Would >>>> be a nice community Project if nothing exists! >>>> >>>> Kind Regards, >>>> Bartek >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected] <javascript:>. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/CAMssQwYa3kW8UMPtJ2PuW8%3Dd8kWB-sz1E99D20ODn28KZTb%2BNQ%40mail.gmail.com >>>> >>>> <https://groups.google.com/d/msgid/prometheus-users/CAMssQwYa3kW8UMPtJ2PuW8%3Dd8kWB-sz1E99D20ODn28KZTb%2BNQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> >>> >>> -- >>> Best Regards, >>> >>> Aliaksandr >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected] <javascript:>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/CAPbKnmBEvDbThrr1cx9FgPhZRVrJqeS2%3DsgEOZguoMWH6FS7BA%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/prometheus-users/CAPbKnmBEvDbThrr1cx9FgPhZRVrJqeS2%3DsgEOZguoMWH6FS7BA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> > > -- > Best Regards, > > Aliaksandr > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9b696103-7d78-4f96-a704-2187adc3219c%40googlegroups.com.
