Hey. I noticed a somewhat unexpected behaviour, perhaps someone can explain why this happens.
- on a Prometheus instance, with a scrape interval of 10s - doing the following queries via curl from the same node where Prometheus runs (so there cannot be any different system times or so Looking at the sample times via e.g.: $ while true; do curl -g 'http://localhost:9090/api/v1/query?query=up[1m]' 2> /dev/null | jq '.data.result[0].values' | grep '[".]' | paste - - ; echo; sleep 1 ; done the timings look suuuuper tight: 1711148768.175, "1" 1711148778.175, "1" 1711148788.175, "1" 1711148798.175, "1" 1711148808.175, "1" 1711148818.175, "1" 1711148768.175, "1" 1711148778.175, "1" 1711148788.175, "1" 1711148798.175, "1" 1711148808.175, "1" 1711148818.175, "1" I.e. it's *always* .175. I guess in reality it may not actually be that tight, and Prometheus just sets the timestamps artificially... but that doesn't matter for me. When now doing a query like (in a while loop with no delay): up[1m] and counting the number of samples, I'd expect to get always either 6 samples or perhaps 7 (if may query would happen exactly at a .175 time). But since the sample times are so super tight, I'd not expect to ever get less than 6. But that's just what happens: 1711148408.137921942 1711148408.179789148 1711148408.190407865 1711148408.239896472 1711148358.175, "1" 1711148368.175, "1" 1711148378.175, "1" 1711148388.175, "1" 1711148398.175, "1" 1711148408.249002352 1711148408.287031384 1711148358.175, "1" 1711148368.175, "1" 1711148378.175, "1" 1711148388.175, "1" 1711148398.175, "1" 1711148408.294628944 1711148408.342150984 1711148408.351871893 1711148408.405270701 Here, the non indented times, are timestamps from before and after the whole curl .. | .. pipe. The indented lines are the samples + timestamps in those cases, where != 6 are returned, done via something similar hacky like: $ while true; do f="$( date +%s.%N >&2; curl -g 'http://localhost:9090/api/v1/query?query=up[1m]' 2> /dev/null | jq '.data.result[0].values' | grep '[".]' | paste - - ; date +%s.%N >&2)"; if [ "$( printf '%s\n' "$f" | wc -l)" != 8 ]; then printf '\n%s\n\n' "$f"; fi ; done One sees, that both times before and after the curl, are already behind the .175, yet still the most recent sample (which should already be there - and which in fact shows up at 1711148408.175 in later queries) is missing. Interestingly, when doing these queries offset 10s (beware that curl requires %20 as space)... none of this happens and I basically always get 6 samples - as more or less expected. [I say more or less, because I wonder,... whether it's possible to get 7 ... should it be?] Any ideas why? And especially also why not with an offset? Thanks, Chris. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0d8c0274-3a29-4b0b-ba06-4d150fe24b39n%40googlegroups.com.
