I recently stumbled upon a ticket that mentioned adding something behind a feature flag, then in Prometheus 3.x making that the default and removing the flag. This approach might work well for changes in behaviour or default configuration values, but it doesn't sound like it would work for flags like ----enable-feature=extra-scrape-metrics and I wonder if those would always stay under --enable-feature flag. I guess there are a few options: - promote it to a dedicated flag - expose those extra metrics under a different path (/metrics/debug ?) so user still needs to opt-in, it's just that the mechanism is different - enable them if /metrics endpoint receives some query params (/metrics?extra-scrape-metrics=true), just a different version of the above - always produce extra metrics and document how to drop it if it's too much
I don't think there's anything wrong with having that under a feature flag, but I do agree that having excessive number of feature flags means that each user runs a slightly different version of prometheus, with a unique set of "features" enabled, which won't improve usability and might be confusing (which flags should I enable and which I shouldn't?). Plus I don't think it's unique to this one use case to have some metrics that are considered "debug" or "optional", so there might be a benefit in some best practice how to expose such metrics when writing instrumentation code. On Friday, 3 September 2021 at 14:11:37 UTC+1 [email protected] wrote: > Having feature flags for extra features that a user needs act on to use > anyway does feel unnecessary from my POV. I feels more like "I acknowledge > this is beta" box tick. > I think it mostly makes sense if one is enabling something that changes > existing behaviour (and so can break existing use cases), possibly with the > assumption that the flag will be removed in the next major release and the > behind-the-flag behaviour would become default. > > I do think flags might also make sense for enabling extra metrics that can > be expensive (like scrape timeouts), mostly because the alternative would > be to have a release notes line announcing new metrics and advising to drop > those via relabelling if not used - this only works when user is upgrading > to that release and if they read notes, for any new install it's likely to > be missed. > > On Friday, 3 September 2021 at 08:44:34 UTC+1 Julien Pivotto wrote: > >> Dear developers, >> >> TL;DR: I'd like to be able to mark feature as experimental in the >> documentation without feature flag. >> >> >> >> >> I am bringing to your attention a discussion we're having in github >> issue https://github.com/prometheus/prometheus/pull/9248. That github >> issue is about having atan2 support in Prometheus as a binary operator. >> >> Because it is strange for users to have atan written as foo atan2 bar, >> instead of atan2(foo, bar), I've asked to mark the feature as >> experimental. Should we in the future add syntactic sugar or add binary >> ops to functions, we would be able to do so. >> >> I also did not want a feature flag for this. In my vision, and that's >> how I acted as maintainer, feature flags should be used when: >> >> - We change an existing behaviour. >> => PromQL query range semantics, with @ delimiter >> => Expanding env variables for external labels >> - We introduce very risky features, that introduce additional memory / >> storage requirements. >> => Remote write receiver >> => Exemplars >> => Snapshots on shutdown >> => Additional scrape metrics (next to up etc) >> >> However, I am not a believer of feature-flags driven Prometheus. >> >> Feature flags have been introduced in Prometheus 2.25. Since then, we >> have added a few new features without feature flag: >> >> - body_size_limit in scrape configs >> - setting timeout and interval via relabeling. >> >> In general, I think it does not benefit users to launch Prometheus with >> lots of feature flags. Our users should be able to assess the risk they >> take by using a feature, without always requiring feature flags. >> Especially for relatively small features like atan2. There is no >> intention to drop atan2 in Prometheus 2.x anyway, just we might find a >> better way to call it. >> >> I try to draw a line between what's a useful feature flag, and where >> just marking experimental in documentation is fine. Prometheus is very >> conservative anyway, and I value the continuity of our features, >> including the "experimental" ones. >> >> Just to give you an idea, if we had a very strong feature flags policy >> in Prometheus, here is what it could have looked like, based on >> >> https://prometheus.io/docs/prometheus/latest/stability/#api-stability-guarantees >> >> >> --enable-feature=promql-at-modifier >> --enable-feature=expand-external-labels >> --enable-feature=promql-negative-offset >> --enable-feature=remote-write-receiver >> --enable-feature=exemplar-storage >> --enable-feature=body-size-limit >> --enable-feature=relabel-intervals >> --enable-feature=remote-read >> --enable-feature=https-basic-auth >> --enable-feature=web-ui >> --enable-feature=service-discovery-k8s >> --enable-feature=service-discovery-consul >> --enable-feature=remote-write-retry-on-429 >> --enable-feature=target-limit >> >> And that's what I want to avoid. >> >> >> Concretely, my proposal is to continue to be able to mark features as >> experimental in the documentation, without requiring feature flags. >> Feature flags can be introduced when some conditions are met, to the >> appreciation of the maintainers. In some cases (breaking changes or >> extra unexpected resource consumption), they are mandatory. >> >> -- >> Julien Pivotto >> @roidelapluie >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/564a3e70-0f3a-44b4-88c4-1a1bc70d49f9n%40googlegroups.com.
