Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-34819

AWS bootstrapping failure due to missing MCS target groups

XMLWordPrintable

    • Critical
    • Yes
    • CLOUD Sprint 254
    • 1
    • Approved
    • False
    • Hide

      None

      Show
      None
    • N/A
    • Release Note Not Required
    • Done

      Some AWS installs are failing to bootstrap due to an issue where CAPA may fail to create load balancer resources, but still declare that infrastructure is ready (see upstream issue for more details).

      In these cases, load balancers are failing to be created due to either rate limiting:

       

      time="2024-05-25T21:43:07Z" level=debug msg="E0525 21:43:07.975223     356 awscluster_controller.go:280] \"failed to reconcile load balancer\" err=<"
      time="2024-05-25T21:43:07Z" level=debug msg="\t[failed to modify target group attribute: Throttling: Rate exceeded" 

      or in some cases another error:

      time="2024-06-01T06:43:58Z" level=debug msg="E0601 06:43:58.902534     356 awscluster_controller.go:280] \"failed to reconcile load balancer\" err=<"
      time="2024-06-01T06:43:58Z" level=debug msg="\t[failed to apply security groups to load balancer \"ci-op-jnqi01di-5feef-92njc-int\": ValidationError: A load balancer ARN must be specified"
      time="2024-06-01T06:43:58Z" level=debug msg="\t\tstatus code: 400, request id: 77446593-03d2-40e9-93c0-101590d150c6, failed to create target group for load balancer: DuplicateTargetGroupName: A target group with the same name 'apiserver-target-1717224237' exists, but with different settings" 

      We have an upstream PR in progress to retry the reconcile logic for load balancers.

       

      Original component readiness report below.

      =====

      Component Readiness has found a potential regression in install should succeed: cluster bootstrap.

      There is no significant evidence of regression

      Sample (being evaluated) Release: 4.16
      Start Time: 2024-05-28T00:00:00Z
      End Time: 2024-06-03T23:59:59Z
      Success Rate: 96.60%
      Successes: 227
      Failures: 8
      Flakes: 0

      Base (historical) Release: 4.15
      Start Time: 2024-02-01T00:00:00Z
      End Time: 2024-02-28T23:59:59Z
      Success Rate: 99.87%
      Successes: 767
      Failures: 1
      Flakes: 0

      View the test details report at https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?arch=amd64&baseEndTime=2024-02-28%2023%3A59%3A59&baseRelease=4.15&baseStartTime=2024-02-01%2000%3A00%3A00&capability=Other&component=Installer%20%2F%20openshift-installer&confidence=95&environment=ovn%20no-upgrade%20amd64%20aws%20standard&excludeArches=arm64%2Cheterogeneous%2Cppc64le%2Cs390x&excludeClouds=openstack%2Cibmcloud%2Clibvirt%2Covirt%2Cunknown&excludeVariants=hypershift%2Cosd%2Cmicroshift%2Ctechpreview%2Csingle-node%2Cassisted%2Ccompact&groupBy=cloud%2Carch%2Cnetwork&ignoreDisruption=true&ignoreMissing=false&minFail=3&network=ovn&pity=5&platform=aws&sampleEndTime=2024-06-03%2023%3A59%3A59&sampleRelease=4.16&sampleStartTime=2024-05-28%2000%3A00%3A00&testId=cluster%20install%3A6ce515c7c732a322333427bf4f5508a5&testName=install%20should%20succeed%3A%20cluster%20bootstrap&upgrade=no-upgrade&variant=standard

              rdossant Rafael Fonseca dos Santos
              padillon Patrick Dillon
              Yunfei Jiang Yunfei Jiang
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Created:
                Updated:
                Resolved: