Yup, this is correct. Prometheus sets the timestamp of the sample at the start of the scrape. But since it's an ACID compliant database, the data is not queryable until after it's been fully ingested.
This is intentional, because the idea is that whatever atomicity is desired by the target is handled by the target. Any locks taken are done when the target receives the GET /metrics. The exposition formatting, compression, and wire transfer time should not impact the "when time when the sample was gathered". And yes, the timing is a tiny bit faked. There are some hidden flags that control this behavior. --scrape.adjust-timestamps --scrape.timestamp-tolerance The default allows up to 2ms (+-0.002) of timing jitter to be ignored. This was added in 2020 due to a regression in the accuracy of the Go internal timer functions. See: https://github.com/prometheus/prometheus/issues/7846 On Sat, Mar 23, 2024 at 10:24 PM Chris Siebenmann < [email protected]> wrote: > > I noticed a somewhat unexpected behaviour, perhaps someone can explain > why > > this happens. > > > > - on a Prometheus instance, with a scrape interval of 10s > > - doing the following queries via curl from the same node where > Prometheus > > runs (so there cannot be any different system times or so > > > > Looking at the sample times via e.g.: > > $ while true; do curl -g ' > http://localhost:9090/api/v1/query?query=up[1m]' > > 2> /dev/null | jq '.data.result[0].values' | grep '[".]' | paste - - ; > > echo; sleep 1 ; done > > > > the timings look suuuuper tight: > > 1711148768.175, "1" > > 1711148778.175, "1" > > 1711148788.175, "1" > > 1711148798.175, "1" > > 1711148808.175, "1" > > 1711148818.175, "1" > > > > 1711148768.175, "1" > > 1711148778.175, "1" > > 1711148788.175, "1" > > 1711148798.175, "1" > > 1711148808.175, "1" > > 1711148818.175, "1" > > > > I.e. it's *always* .175. I guess in reality it may not actually be that > > tight, and Prometheus just sets the timestamps artificially... but that > > doesn't matter for me. > > I think that it does potentially matter for you in this situation. My > understanding is that Prometheus freezes the timestamp for all ingested > samples from a particular scrape at the time where the scrape begins > (including the 'up{}' sample). However, I wouldn't expect the samples to > appear in the TSDB and be available for queries until after the scrape > finishes, and that might take some amount of time (you can see this in > the scrape_duration_seconds metric). > > (The timestamps are so exact because it is ten seconds from the start of > the previous scrape, not its end. As long as scrapes take less than ten > seconds, you'll see them start on this extremely even timeline.) > > So if you query for live data as of 'right now' and do so often enough > and fast enough, I believe that you may miss a scrape that is currently > in progress and being ingested into the TSDB. If you offset your query > enough into the past that the scrape and ingestion will have finished, > you will reliably get the results you expect. > > - cks > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/3658428.1711229055%40apps0.cs.toronto.edu > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmq6QWfekOOvYcf6LFhWVbzvSib8q3B0H-AT5fdf_xfqGQ%40mail.gmail.com.
