Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-12083

Thanos is not able to connect to S3 bucket.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Normal Normal
    • None
    • ACM 2.10.Z
    • Observability
    • False
    • None
    • False
    • Important
    • Customer Facing, Customer Reported
    • -
    • No

      Description of problem:

      Case: https://access.redhat.com/support/cases/#/case/03835223 

      Thread:  https://redhat-internal.slack.com/archives/CUU609ZQC/p1717744496503429 

      Thanos is not able to connect S3-compatible storage with error:

       

      Service Unavailable
      

      Version-Release number of selected component (if applicable):

      ACM 2.10

      How reproducible:

      Always

      Steps to Reproduce:

      1.  Enable observability
      2.  Set storage
      3. ...

      Actual results:

      Expected results:

      Additional info:

      ts=2024-06-07T13:35:13.089531276Z caller=factory.go:53 level=info msg="loading bucket configuration"
      ts=2024-06-07T13:36:28.137321856Z caller=main.go:135 level=error err="Get \"https://storage.server/bucketname/?location=\": Service Unavailable\nBaseFetcher: iter bucket\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetchMetadata\n\t/remote-source/thanos/app/pkg/block/fetcher.go:383\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetch.func2\n\t/remote-source/thanos/app/pkg/block/fetcher.go:456\ngithub.com/golang/groupcache/singleflight.(*Group).Do\n\t/remote-source/thanos/deps/gomod/pkg/mod/github.com/golang/groupcache@v0.0.0-20210331224755-41bb18bfe9da/singleflight/singleflight.go:56\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetch\n\t/remote-source/thanos/app/pkg/block/fetcher.go:454\ngithub.com/thanos-io/thanos/pkg/block.(*MetaFetcher).Fetch\n\t/remote-source/thanos/app/pkg/block/fetcher.go:514\nmain.registerBucketInspect.func1\n\t/remote-source/thanos/app/cmd/thanos/tools_bucket.go:522\nmain.main\n\t/remote-source/thanos/app/cmd/thanos/main.go:133\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1650\npreparing tools bucket inspect command failed\nmain.main\n\t/remote-source/thanos/app/cmd/thanos/main.go:135\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1650"

       

      Storage is accessible from Thanos pods, tested with openssl, AWS cli... Certs are valid (tested with openssl/aws clii too).

      More in private comment.

              Unassigned Unassigned
              rhn-support-mheppler Michal Heppler
              Xiang Yin Xiang Yin
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: