Hello,
I am getting started on "*Solution to Problem 1:*". I will share the PR
here once it's ready for review.
Thanks,
Ganesh
On Wednesday, 30 March 2022 at 13:38:01 UTC+5:30 Ganesh Vernekar wrote:
> Hello everyone,
>
> I have come across hosted metrics providers (including Grafana Cloud that
> I am personally looking at) facing issues in being compliant with alert
> delivery. Here are the main problems that I identify:
>
> *Problem 1:* They have some kind of embedded alert routing mechanism
> ("alertmanager") which you cannot bypass to send directly to external
> alertmanagers. Allowing external alertmanagers is not trivial and also has
> security implications. *So the alert payload that it sends might not
> exactly match what the test suite is expecting.*
>
> *Problem 2:* Related to the above, the alert routing mechanism *can add
> unwanted delays in sending alerts or reflecting the changes in the alert
> annotations.*
>
> Here is what I am proposing for the above problems:
>
> *Solution to Problem 1:* Allow cloud providers to have custom unmarshal
> logic for the alert payload embedded in the test suite. For example,
> currently we require the alert payload to be directly unmarshalled into
> []notifier.Alert here
> <https://github.com/prometheus/compliance/blob/c7c726de89973d77cb491faa1b32cfddf7dcde8a/alert_generator/server.go#L91>.
>
> But now we can allow embedding of custom unmarshalling logic such that at
> the end of the custom logic, it provides *[]notifier.Alert* as the result
> to the test suite to verify.
>
> *Solution to Problem 2:* It is in the hands of cloud providers to take
> care of it, to allow forwarding of the alerts to the end receiver with
> minimal delays and to forward all the alerts. For those who use upstream
> Prometheus Alertmanager, we need support from upstream since Prometheus
> Alertmanager does not allow forwarding of all alerts that it gets. For that
> I have requested a feature here
> <https://github.com/prometheus/alertmanager/issues/2868>.
>
> Please let me know if you have any better suggestions and/or any
> objections to the proposed solutions and/or +1 to the proposed solutions.
>
> Thanks,
> Ganesh
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-developers/4d7eef8c-76be-406b-862a-49e3ed61873dn%40googlegroups.com.
