Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-35448

AWS bootstrapping failure due to missing MCS target groups

XMLWordPrintable

    • Critical
    • Yes
    • Approved
    • False
    • Hide

      None

      Show
      None
    • N/A
    • Release Note Not Required
    • In Progress

      This is a clone of issue OCPBUGS-34819. The following is the description of the original issue:

      Some AWS installs are failing to bootstrap due to an issue where CAPA may fail to create load balancer resources, but still declare that infrastructure is ready (see upstream issue for more details).

      In these cases, load balancers are failing to be created due to either rate limiting:

       

      time="2024-05-25T21:43:07Z" level=debug msg="E0525 21:43:07.975223     356 awscluster_controller.go:280] \"failed to reconcile load balancer\" err=<"
      time="2024-05-25T21:43:07Z" level=debug msg="\t[failed to modify target group attribute: Throttling: Rate exceeded" 

      or in some cases another error:

      time="2024-06-01T06:43:58Z" level=debug msg="E0601 06:43:58.902534     356 awscluster_controller.go:280] \"failed to reconcile load balancer\" err=<"
      time="2024-06-01T06:43:58Z" level=debug msg="\t[failed to apply security groups to load balancer \"ci-op-jnqi01di-5feef-92njc-int\": ValidationError: A load balancer ARN must be specified"
      time="2024-06-01T06:43:58Z" level=debug msg="\t\tstatus code: 400, request id: 77446593-03d2-40e9-93c0-101590d150c6, failed to create target group for load balancer: DuplicateTargetGroupName: A target group with the same name 'apiserver-target-1717224237' exists, but with different settings" 

      We have an upstream PR in progress to retry the reconcile logic for load balancers.

       

      Original component readiness report below.

      =====

      Component Readiness has found a potential regression in install should succeed: cluster bootstrap.

      There is no significant evidence of regression

      Sample (being evaluated) Release: 4.16
      Start Time: 2024-05-28T00:00:00Z
      End Time: 2024-06-03T23:59:59Z
      Success Rate: 96.60%
      Successes: 227
      Failures: 8
      Flakes: 0

      Base (historical) Release: 4.15
      Start Time: 2024-02-01T00:00:00Z
      End Time: 2024-02-28T23:59:59Z
      Success Rate: 99.87%
      Successes: 767
      Failures: 1
      Flakes: 0

      View the test details report at https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?arch=amd64&baseEndTime=2024-02-28%2023%3A59%3A59&baseRelease=4.15&baseStartTime=2024-02-01%2000%3A00%3A00&capability=Other&component=Installer%20%2F%20openshift-installer&confidence=95&environment=ovn%20no-upgrade%20amd64%20aws%20standard&excludeArches=arm64%2Cheterogeneous%2Cppc64le%2Cs390x&excludeClouds=openstack%2Cibmcloud%2Clibvirt%2Covirt%2Cunknown&excludeVariants=hypershift%2Cosd%2Cmicroshift%2Ctechpreview%2Csingle-node%2Cassisted%2Ccompact&groupBy=cloud%2Carch%2Cnetwork&ignoreDisruption=true&ignoreMissing=false&minFail=3&network=ovn&pity=5&platform=aws&sampleEndTime=2024-06-03%2023%3A59%3A59&sampleRelease=4.16&sampleStartTime=2024-05-28%2000%3A00%3A00&testId=cluster%20install%3A6ce515c7c732a322333427bf4f5508a5&testName=install%20should%20succeed%3A%20cluster%20bootstrap&upgrade=no-upgrade&variant=standard

              rdossant Rafael Fonseca dos Santos
              openshift-crt-jira-prow OpenShift Prow Bot
              Yunfei Jiang Yunfei Jiang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: