That is a very good question (: I don't think the current API of client Golang https://godoc.org/github.com/prometheus/client_golang/prometheus/testutil package allows this scenario. It might be good to add GH issue for client_golang project to explore those possibilities,
In terms of potential API for such util functions, something that I would recommend here is the e2e framework we use in Thanos like here: https://github.com/thanos-io/thanos/blob/c733564d44745af1a023bfa5d51d6d205404dc82/test/e2e/compact_test.go#L556 This is actually maintained in Cortex repository, and it's something we use, maintain and recommend, especially if you want to test against container environment. There is also a possibility to pull out this code to operate on raw text files, and put into client_golang, this API might work. Note that Peter also is working on something even better: PromQL based unit test on metric text file! <https://github.com/cortexproject/cortex/pull/2524> (: Quite amazing stuff, something to consider as well. Kind Regards, Bartek On Thu, 28 May 2020 at 19:59, 'Michael Rybak' via Prometheus Developers < [email protected]> wrote: > *TL;DR: I'm unit-testing Prometheus metrics: server's requests count and > latencies. The server is written in Go. Latencies are non-determinstic and > there seems to be no good way to fake clock in Prometheus or Go. Does > Prometheus provide a method similar to CollectAndCompare that lets > validating a value (e.g. latency) against a range, not exact expectation? > If not, would it be a good feature to add to Prometheus? Alternatively, is > there a way to fake clock in Prometheus, so > e.g. InstrumentHandlerDuration would propagate the pre-defined fake > duration? Or, is there a good way to fake clock in Go?* > > I'm measuring server's requests count (Counter) and latencies (Summary), > and test both with unit tests. I observe the metrics via > InstrumentHandlerCounter and InstrumentHandlerDuration respectively. > > For testing requests count, I hardcode the expectation in a string > constant, and use CollectAndCompare to perform an exact match validation: > > // Make some test requests. > .. > > // Validate them > expectation := strings.NewReader(` > # HELP my_metric_requests_total My help. > # TYPE my_metric_requests_total counter > my_metric_requests_total{code="200"} 2 > my_metric_requests_total{code="304"} 1 > my_metric_requests_total{code="502"} 1 > my_metric_requests_total{code="503"} 1 > `) > > this.Require().NoError(promtest.CollectAndCompare(myMetricRequestsTotal, > expectation, "my_metric_requests_total")) > > I couldn't find a way to do the same for latencies, because they are > non-deterministic. So instead of the one-liner check above I dig into the > internals of the gathered metrics: > > // Make some test requests. > hintPrefix := "My test." > ... > > // Validate them > type codeLabelPair string > type scenarioExpectedSampleCountMap map[codeLabelPair]uint64 > > expectedSampleCountMap := scenarioExpectedSampleCountMap{ > `name:"code" value:"200" `: 3, > `name:"code" value:"304" `: 1, > `name:"code" value:"502" `: 2, > } > > reg := prometheus.NewPedanticRegistry() > if err := reg.Register(promRequestsLatency); err != nil { > this.T().Errorf(hintPrefix+" - registering collector failed: %s", err) > } > > actualMetricFamilyArr, err := reg.Gather() > if err != nil { > this.T().Errorf(hintPrefix+" - gathering metrics failed: %s", err) > } > > assert.Equal(this.T(), 1, len(actualMetricFamilyArr), > hintPrefix+" expects exactly one metric family.") > > assert.Equal(this.T(), "request_latencies_in_seconds", > *actualMetricFamilyArr[0].Name, > hintPrefix+" expects the right metric name.") > > assert.Equal(this.T(), len(expectedSampleCountMap), > len(actualMetricFamilyArr[0].Metric), > hintPrefix+" expects the right amount of metrics collected and gathered.") > > for _, actualMetric := range actualMetricFamilyArr[0].Metric { > // Expect the right sample count. > code := actualMetric.Label[0].String() > expectedSampleCount := expectedSampleCountMap[codeLabelPair(code)] > actualSampleCount := actualMetric.Summary.GetSampleCount() > assert.Equal(this.T(), expectedSampleCount, actualSampleCount, > hintPrefix+" expects the right sample count for "+code) > > // Test quantiles. > expectedQuantileKeys := []float64{0.5, 0.9, 0.99} > > // Expect the right number of quantiles. > assert.Equal(this.T(), len(expectedQuantileKeys), > len(actualMetric.Summary.Quantile), hintPrefix+" expects the right number > of quantiles.") > > // Expect the right quantiles. > // Expect positive quantile values, because latencies are non-zero. > // Don't check the exact values, because latencies are non-deterministic. > for i, quantile := range actualMetric.Summary.Quantile { > assert.Equal(this.T(), expectedQuantileKeys[i], quantile.GetQuantile(), > hintPrefix+" expects the right quantile.") > assert.True(this.T(), quantile.GetValue() > .0, hintPrefix+" expects > non-zero quantile value (latency).") > } > } > > This seems to be more complex than it should be. Is there a one-liner way, > similar to the CollectAndCompare call I'm making above to validate > requests count? > > Alternatively, is there a way to fake clock in Prometheus, so e.g. > InstrumentHandlerDuration would propagate the pre-defined fake duration? > Or, is there a good way to fake clock in Go? There is an option that > doesn't look safe enough: > > https://www.reddit.com/r/golang/comments/30try1/monkey_patching_in_go/ > https://news.ycombinator.com/item?id=22442170. > > Thanks. > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/b440c496-cfb1-4b5f-8fb5-11ca47485a3cn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-developers/b440c496-cfb1-4b5f-8fb5-11ca47485a3cn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAMssQwY%2BsHEjeejfG3%3DXBJMSYCQL7YOWW0vBnT_8z_Ru0hxJEg%40mail.gmail.com.
