The "is prometheus running" msg was a red herring. The real issue was that err shown next. After setting storage.tsdb.max-block-duration and storage.tsdb.max-block-duration to 2h, I was able to run sidecar.
On Wednesday, September 4, 2024 at 1:58:30 PM UTC-7 mel wrote: > Hello I am trying to run thanos sidecar on a on-prem prometheus instance > to ship logs to AWS S3, and I will run thanos in AWS. My understanding is > that this should work > > # thanos sidecar --tsdb.path /data/prometheus/ --prometheus.url > http://localhost:9090 --objstore.config-file > /etc/prometheus/bucket_config.yaml > > But I get this error > > ts=2024-09-04T20:49:45.622990448Z caller=options.go:26 level=info > protocol=gRPC msg="disabled TLS, key and cert must be set to enable" > ts=2024-09-04T20:49:45.623495042Z caller=factory.go:53 level=info > msg="loading bucket configuration" > ts=2024-09-04T20:49:45.623870275Z caller=sidecar.go:426 level=info > msg="starting sidecar" > ts=2024-09-04T20:49:45.624053759Z caller=reloader.go:262 level=info > component=reloader msg="nothing to be watched" > ts=2024-09-04T20:49:45.624159982Z caller=intrumentation.go:75 level=info > msg="changing probe status" status=healthy > ts=2024-09-04T20:49:45.624263372Z caller=http.go:73 level=info > service=http/server component=sidecar msg="listening for requests and > metrics" address=0.0.0.0:10902 > ts=2024-09-04T20:49:45.624543225Z caller=tls_config.go:313 level=info > service=http/server component=sidecar msg="Listening on" address=[::]:10902 > ts=2024-09-04T20:49:45.625111798Z caller=tls_config.go:316 level=info > service=http/server component=sidecar msg="TLS is disabled." http2=false > address=[::]:10902 > ts=2024-09-04T20:49:45.626098488Z caller=sidecar.go:193 level=warn > msg="failed to validate prometheus flags. Is Prometheus running? Retrying" > err="found that TSDB Max time is 1d12h and Min time is 2h. Compaction needs > to be disabled (storage.tsdb.min-block-duration = > storage.tsdb.max-block-duration)" > ^Cts=2024-09-04T20:49:47.877678349Z caller=main.go:182 level=info > msg="caught signal. Exiting." signal=interrupt > ts=2024-09-04T20:49:47.877756234Z caller=intrumentation.go:67 level=warn > msg="changing probe status" status=not-ready reason=null > ts=2024-09-04T20:49:47.877774686Z caller=http.go:91 level=info > service=http/server component=sidecar msg="internal server is shutting > down" err=null > ts=2024-09-04T20:49:47.877876118Z caller=http.go:110 level=info > service=http/server component=sidecar msg="internal server is shutdown > gracefully" err=null > ts=2024-09-04T20:49:47.877982816Z caller=intrumentation.go:81 level=info > msg="changing probe status" status=not-healthy reason=null > ts=2024-09-04T20:49:47.87814256Z caller=grpc.go:174 level=info > service=gRPC/server component=sidecar msg="internal server is shutting > down" err=null > ts=2024-09-04T20:49:47.878219725Z caller=grpc.go:187 level=info > service=gRPC/server component=sidecar msg="gracefully stopping internal > server" > ts=2024-09-04T20:49:47.878326514Z caller=grpc.go:200 level=info > service=gRPC/server component=sidecar msg="internal server is shutdown > gracefully" err=null > ts=2024-09-04T20:49:47.878405123Z caller=main.go:174 level=info msg=exiting > > Prometheus is definitely running > > # prometheus --version > prometheus, version 2.47.2 (branch: HEAD, revision: > 3f3172cde1ee37f1c7b3a5f3d9b031190509b3ad) > build user: root@79f2ad339b75 > build date: 20231012-16:07:10 > go version: go1.21.3 > platform: linux/amd64 > tags: netgo,builtinassets,stringlabels > > > > # curl http://localhost:9090 > <a href="/graph">Found</a>. > > > > # ls /data/prometheus > 01J5SMXQ9NH8YFQHBGQ3MNS6J3 01J5ZEAABQQ5FKN10M497V9PY9 > 01J657Q89PQ2VG1Q4ECAT5G5YZ 01J6B13VKC5WZ0YZ30VH6Z8ES9 > 01J6GTG80PTKBQ1A4H5T5VY9HC 01J6PKWV8RZDDHTNFZQBYG3RZS > 01J6WD9S20Z5Q6HS9N5BS40DMJ 01J6YZN4MMM7ZM5W9NXFEMZ9MT chunks_head wal > 01J5VJQ8662PZNG6HBTTP1MC8B 01J61C451ERBCVBA53BAJRPQJM > 01J675GSSK1ZVQ8NXWT8DKZQ4S 01J6CYXCVCYY9XVE9TZAXTKX5X > 01J6JRA0XYMJE7CE9P5ZKF6TTV 01J6RHPNFX8XKMM29M2TN3FN1M > 01J6YB31NZTV4J4N9C3PVG8487 01J6YZNQN83DAM0H1BVYC5A2KC lock > 01J5XGGSA7A4WXT252HSRQJ3PC 01J639XPHKXAEGWGYS1CSS5NF0 > 01J693AB1A89Y7G5MRBZ6SBZT5 01J6EWPYS460286NA24MN5NC2T > 01J6MP3FETWR1GJV3N6M0FHK01 01J6TFG4RD0KQKEK2RM220N1XZ > 01J6YRSDCMDJDM2XN91MAN58QY 01J6Z6GVWNNHPNX3C1DKRN5GWR queries.active -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8096ada3-00fa-4794-bcf7-37b5443bb8b2n%40googlegroups.com.
