Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-28535

CCO Pod crashes on BM cluster when AWS Root Credential exists

XMLWordPrintable

    • No
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, creating an {aws-first} root secret on a bare metal cluster caused the Cloud Credential Operator (CCO) pod to crash.
      The issue is resolved in this release.
      (link:https://issues.redhat.com/browse/OCPBUGS-28535[*OCPBUGS-28535*])
      Show
      * Previously, creating an {aws-first} root secret on a bare metal cluster caused the Cloud Credential Operator (CCO) pod to crash. The issue is resolved in this release. (link: https://issues.redhat.com/browse/OCPBUGS-28535 [* OCPBUGS-28535 *])
    • Bug Fix
    • Done

      Description of problem:

      Similar to https://bugzilla.redhat.com/show_bug.cgi?id=1996624, when the AWS root credential (must possesses the "iam:SimulatePrincipalPolicy" permission) exists on a BM cluster, the CCO Pod crashes when running the secretannotator controller. 

      Steps to Reproduce:

      1. Install a BM cluster
      fxie-mac:cloud-credential-operator fxie$ oc get infrastructures.config.openshift.io cluster -o yaml
      apiVersion: config.openshift.io/v1
      kind: Infrastructure
      metadata:
        creationTimestamp: "2024-01-28T19:50:05Z"
        generation: 1
        name: cluster
        resourceVersion: "510"
        uid: 45bc2a29-032b-4c74-8967-83c73b0141c4
      spec:
        cloudConfig:
          name: ""
        platformSpec:
          type: None
      status:
        apiServerInternalURI: https://api-int.fxie-bm1.qe.devcluster.openshift.com:6443
        apiServerURL: https://api.fxie-bm1.qe.devcluster.openshift.com:6443
        controlPlaneTopology: SingleReplica
        cpuPartitioning: None
        etcdDiscoveryDomain: ""
        infrastructureName: fxie-bm1-x74wn
        infrastructureTopology: SingleReplica
        platform: None
        platformStatus:
          type: None 
      
      2. Create an AWS user with IAMReadOnlyAccess permissions:
      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Action": [
                      "iam:GenerateCredentialReport",
                      "iam:GenerateServiceLastAccessedDetails",
                      "iam:Get*",
                      "iam:List*",
                      "iam:SimulateCustomPolicy",
                      "iam:SimulatePrincipalPolicy"
                  ],
                  "Resource": "*"
              }
          ]
      }
      
      3. Create AWS root credentials with a set of access keys of the user above
      4. Trigger a reconcile of the secretannotator controller, e.g. via editting cloudcredential/cluster     

      Logs:

      time="2024-01-29T04:47:27Z" level=warning msg="Action not allowed with tested creds" action="iam:CreateAccessKey" controller=secretannotator
      time="2024-01-29T04:47:27Z" level=warning msg="Action not allowed with tested creds" action="iam:CreateUser" controller=secretannotator
      time="2024-01-29T04:47:27Z" level=warning msg="Action not allowed with tested creds" action="iam:DeleteAccessKey" controller=secretannotator
      time="2024-01-29T04:47:27Z" level=warning msg="Action not allowed with tested creds" action="iam:DeleteUser" controller=secretannotator
      time="2024-01-29T04:47:27Z" level=warning msg="Action not allowed with tested creds" action="iam:DeleteUserPolicy" controller=secretannotator
      time="2024-01-29T04:47:27Z" level=warning msg="Action not allowed with tested creds" action="iam:PutUserPolicy" controller=secretannotator
      time="2024-01-29T04:47:27Z" level=warning msg="Action not allowed with tested creds" action="iam:TagUser" controller=secretannotator
      time="2024-01-29T04:47:27Z" level=warning msg="Tested creds not able to perform all requested actions" controller=secretannotator
      I0129 04:47:27.988535       1 reflector.go:289] Starting reflector *v1.Infrastructure (10h37m20.569091933s) from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:233
      I0129 04:47:27.988546       1 reflector.go:325] Listing and watching *v1.Infrastructure from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:233
      I0129 04:47:27.989503       1 reflector.go:351] Caches populated for *v1.Infrastructure from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:233
      panic: runtime error: invalid memory address or nil pointer dereference [recovered]
      panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1a964a0]
       
      goroutine 341 [running]:
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
      /go/src/github.com/openshift/cloud-credential-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:115 +0x1e5
      panic({0x3fe72a0?, 0x809b9e0?})
      /usr/lib/golang/src/runtime/panic.go:914 +0x21f
      github.com/openshift/cloud-credential-operator/pkg/operator/utils/aws.LoadInfrastructureRegion({0x562e1c0?, 0xc002c99a70?}, {0x5639ef0, 0xc0001b6690})
      /go/src/github.com/openshift/cloud-credential-operator/pkg/operator/utils/aws/utils.go:72 +0x40
      github.com/openshift/cloud-credential-operator/pkg/operator/secretannotator/aws.(*ReconcileCloudCredSecret).validateCloudCredsSecret(0xc0008c2000, 0xc002586000)
      /go/src/github.com/openshift/cloud-credential-operator/pkg/operator/secretannotator/aws/reconciler.go:206 +0x1a5
      github.com/openshift/cloud-credential-operator/pkg/operator/secretannotator/aws.(*ReconcileCloudCredSecret).Reconcile(0xc0008c2000, {0x30?, 0xc000680c00?}, {0x4f38a3d?, 0x0?}, {0x4f33a20?, 0x416325?})
      /go/src/github.com/openshift/cloud-credential-operator/pkg/operator/secretannotator/aws/reconciler.go:166 +0x605
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x561ff20?, {0x561ff20?, 0xc002ff3b00?}, {0x4f38a3d?, 0x3b180c0?}, {0x4f33a20?, 0x55eea08?})
      /go/src/github.com/openshift/cloud-credential-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118 +0xb7
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000189360, {0x561ff58, 0xc0007e5040}, {0x4589f00?, 0xc000570b40?})
      /go/src/github.com/openshift/cloud-credential-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:314 +0x365
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000189360, {0x561ff58, 0xc0007e5040})
      /go/src/github.com/openshift/cloud-credential-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265 +0x1c9
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
      /go/src/github.com/openshift/cloud-credential-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226 +0x79
      created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 183
      /go/src/github.com/openshift/cloud-credential-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222 +0x565

      Actual results:

      CCO Pod crashes and restarts in a loop:
      fxie-mac:cloud-credential-operator fxie$ oc get po -n openshift-cloud-credential-operator -w
      NAME                                         READY   STATUS    RESTARTS        AGE
      cloud-credential-operator-657bdffdff-9wzrs   2/2     Running   3 (2m35s ago)   8h

              fxierh Feilian Xie
              fxierh Feilian Xie
              Jianping Shu Jianping Shu
              Jeana Routh Jeana Routh
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: