Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-59251

aws-cloud-controller-manager runs into failure after cluster upgrade due to 'cloud-conf' ConfigMap is missing

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • CLOUD Sprint 274, CLOUD Sprint 275, CLOUD Sprint 276
    • 3
    • Done
    • Bug Fix
    • Hide
      * Before this update, {product-title} clusters on {aws-short} that were created with version 4.13 or earlier could not update to version 4.19. Clusters that were created with version 4.14 and later have an {aws-short} `cloud-conf` ConfigMap by default, and this ConfigMap is required starting in {product-title} 4.19. With this release, the Cloud Controller Manager Operator is updated to create a default `cloud-conf` ConfigMap when none is present on the cluster. This change enables clusters that were created with version 4.13 or earlier to update to version 4.19. link:https://issues.redhat.com/browse/OCPBUGS-59251[OCPBUGS-59251]
      Show
      * Before this update, {product-title} clusters on {aws-short} that were created with version 4.13 or earlier could not update to version 4.19. Clusters that were created with version 4.14 and later have an {aws-short} `cloud-conf` ConfigMap by default, and this ConfigMap is required starting in {product-title} 4.19. With this release, the Cloud Controller Manager Operator is updated to create a default `cloud-conf` ConfigMap when none is present on the cluster. This change enables clusters that were created with version 4.13 or earlier to update to version 4.19. link: https://issues.redhat.com/browse/OCPBUGS-59251 [ OCPBUGS-59251 ]
    • None
    • None
    • None
    • None

      Description of problem:

      The `cloud-conf` ConfigMap in the `openshift-cloud-controller-manager` namespace does not exist:

      $ oc get cm cloud-conf -n openshift-cloud-controller-manager
Error from server (NotFound): configmaps "cloud-conf" not found

      Causing the `aws-cloud-controller-manager` pods to fail:

      oc describe pod aws-cloud-controller-manager-xxx-xxxx -n openshift-cloud-controller-manager
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    90m                  default-scheduler  Successfully assigned openshift-cloud-controller-manager/aws-cloud-controller-manager-xxx-xxxx to ip-xx-xxx-xxx-xxx.ec2.internal
  Warning  FailedMount  5m3s (x50 over 90m)  kubelet            MountVolume.SetUp failed for volume "config-accm" : configmap "cloud-conf" not found

      This is caused by an empty value within the infrastructure file, as no value results in it not being synced:

      $ oc get infrastructures.config.openshift.io -oyaml | yq '.items[].spec.cloudConfig'
name: ""

      See also linked below the change likely to originate the issue.

       

      Version-Release number of selected component (if applicable):

      4.19+

      How reproducible:

          Appears to occur in 'candidate' channel only as of current.

      Steps to Reproduce:

          1. Deploy/upgrade non-CCS cluster to 4.19
    2. Observe missing 'cloud-conf' configmap in `openshift-cloud-controller-manager` namespace.

      Actual results:

      No `cloud-conf` configmap -> Infrastructure cloudconfig value empty -> controller pods failing to create   

      Expected results:

       

      Additional info:

      Relevant code: https://github.com/openshift/cluster-cloud-controller-manager-operator/commit/c255a75e95434cfc5d1fd73b6d0399ede413616e

CCS docs: https://docs.redhat.com/en/documentation/openshift_dedicated/4/html/planning_your_environment/aws-ccs

              rh-ee-nbrubake Nolan Brubaker
              msaary.openshift Mate Saary
              None
              None
              Huali Liu Huali Liu
              None
              Votes:
              1 Vote for this issue
              Watchers:
              20 Start watching this issue

                Created:
                Updated: