That's the TSDB complaining about a related problem where multiple series with the same label identity are likely coming in. But that shouldn't lead to PromQL queries receiving the same series multiple times in a vector selector unless something is broken in the TSDB / Thanos data merging layer underneath PromQL. Could you share all the details of your Thanos setup (how many Prometheus servers with which external labels do you have, how are you merging them together with Thanos, which Thanos and Prometheus versions are you using, are you using object storage for long-term data, etc.)? Might at some point be a better question / bug report for the Thanos folks though.
On Wed, May 31, 2023 at 3:07 PM Adso “Cho” Castro <[email protected]> wrote: > Hey Julius, morning > > Found a few warn logs from thanos-receive, maybe a possible root cause? > > - "Error on ingesting samples with different values but same timestamp" > - "Error on ingesting out-of-order samples" > > > > Em terça-feira, 30 de maio de 2023 às 18:02:26 UTC-3, Julius Volz escreveu: > >> Nothing in the query itself should be able to cause a duplicate >> labelset in a normal PromQL execution, so I assume that something is wrong >> with the underlying Thanos setup or code. Maybe it somehow manages to >> select the same duplicate time series from multiple Prometheus servers (or >> the long-term storage), because something around the external labels >> normally used for differentiating series isn't configured or working >> properly? >> >> On Tue, May 30, 2023 at 10:28 PM Adso “Cho” Castro <[email protected]> >> wrote: >> >>> Hey, been getting the error below when querying: >>> >>> *execution: vector cannot contain metrics with the same labelset* >>> >>> The query: >>> >>> sum(rate(istio_requests_total{reporter="source", >>> destination_workload="dummy-data", response_code=~"2.."}[$_range])) by >>> (destination_workload) / sum(rate(istio_requests_total{reporter="source", >>> destination_workload="dummy-data"}[$_range])) by (destination_workload) >>> >>> Funny thing is, this is a Thanos installation. I have a Cortex >>> installation with the same data and this query works fine. >>> >>> Thanks >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/b08f43a8-fd11-4775-838e-350f144df8e8n%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/b08f43a8-fd11-4775-838e-350f144df8e8n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Julius Volz >> PromLabs - promlabs.com >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/36531430-934f-4334-a78c-66abb1b16aa7n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/36531430-934f-4334-a78c-66abb1b16aa7n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Julius Volz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5xrSvT5%3DxNDbW%2B4Ym27%2BVVo%2B7dGiXcF1B1RJsWtLEQ%2BHg%40mail.gmail.com.
