-
Bug
-
Resolution: Done
-
Normal
-
None
-
ACM 2.10.Z
-
False
-
None
-
False
-
-
-
Important
-
Customer Facing, Customer Reported
-
-
-
No
Description of problem:
Case: https://access.redhat.com/support/cases/#/case/03835223
Thread: https://redhat-internal.slack.com/archives/CUU609ZQC/p1717744496503429
Thanos is not able to connect S3-compatible storage with error:
Service Unavailable
Version-Release number of selected component (if applicable):
ACM 2.10
How reproducible:
Always
Steps to Reproduce:
- Enable observability
- Set storage
- ...
Actual results:
Expected results:
Additional info:
ts=2024-06-07T13:35:13.089531276Z caller=factory.go:53 level=info msg="loading bucket configuration" ts=2024-06-07T13:36:28.137321856Z caller=main.go:135 level=error err="Get \"https://storage.server/bucketname/?location=\": Service Unavailable\nBaseFetcher: iter bucket\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetchMetadata\n\t/remote-source/thanos/app/pkg/block/fetcher.go:383\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetch.func2\n\t/remote-source/thanos/app/pkg/block/fetcher.go:456\ngithub.com/golang/groupcache/singleflight.(*Group).Do\n\t/remote-source/thanos/deps/gomod/pkg/mod/github.com/golang/groupcache@v0.0.0-20210331224755-41bb18bfe9da/singleflight/singleflight.go:56\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetch\n\t/remote-source/thanos/app/pkg/block/fetcher.go:454\ngithub.com/thanos-io/thanos/pkg/block.(*MetaFetcher).Fetch\n\t/remote-source/thanos/app/pkg/block/fetcher.go:514\nmain.registerBucketInspect.func1\n\t/remote-source/thanos/app/cmd/thanos/tools_bucket.go:522\nmain.main\n\t/remote-source/thanos/app/cmd/thanos/main.go:133\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1650\npreparing tools bucket inspect command failed\nmain.main\n\t/remote-source/thanos/app/cmd/thanos/main.go:135\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1650"
Storage is accessible from Thanos pods, tested with openssl, AWS cli... Certs are valid (tested with openssl/aws clii too).
More in private comment.