Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-14120

Replacing master node on GCP IPI don't include new machine into the instance-group used for api-int lb

    XMLWordPrintable

Details

    • No
    • False
    • Hide

      None

      Show
      None

    Description

      Description of problem:

      - IPI cluster on GCP 
      - Replaced a master node following the proccedure described here In Replacing an unhealthy etcd member whose machine is not running or whose node is not ready https://docs.openshift.com/container-platform/4.12/backup_and_restore/control_plane_backup_and_restore/replacing-unhealthy-etcd-member.html
      - Provisioning the new machine using machine-api doesn't include the new provisioned virtual machine into the instance-group which is used for the endpoints of the API-INT load balancer at cloud provider

      Version-Release number of selected component (if applicable):

       

      How reproducible:

      - Create a new master machine
      $ cat new-master0-machine.yaml
      apiVersion: machine.openshift.io/v1beta1
      kind: Machine
      metadata:
        annotations:
        finalizers:
        - machine.machine.openshift.io
        generation: 3
        labels:
          machine.openshift.io/cluster-api-cluster: case-03507533-rfctg
          machine.openshift.io/cluster-api-machine-role: master
          machine.openshift.io/cluster-api-machine-type: master
          machine.openshift.io/instance-type: n2-standard-4
          machine.openshift.io/region: europe-southwest1
          machine.openshift.io/zone: europe-southwest1-a
        name: case-03507533-rfctg-master-0-v2
        namespace: openshift-machine-api
      spec:
        lifecycleHooks:
          preDrain:
          - name: EtcdQuorumOperator
            owner: clusteroperator/etcd
        metadata: {}
        providerSpec:
          value:
            apiVersion: machine.openshift.io/v1beta1
            canIPForward: false
            credentialsSecret:
              name: gcp-cloud-credentials
            deletionProtection: false
            disks:
            - autoDelete: true
              boot: true
              image: projects/rhcos-cloud/global/images/rhcos-412-86-202303211731-0-gcp-x86-64
              labels: null
              sizeGb: 128
              type: pd-ssd
            kind: GCPMachineProviderSpec
            machineType: n2-standard-8
            metadata:
              creationTimestamp: null
            networkInterfaces:
            - network: case-03507533-rfctg-network
              subnetwork: case-03507533-rfctg-master-subnet
            projectID: cee-gcp-emea
            region: europe-southwest1
            serviceAccounts:
            - email: xxxxxxxxx
              scopes:
              - https://www.googleapis.com/auth/cloud-platform
            tags:
            - case-03507533-rfctg-master
            targetPools:
            - case-03507533-rfctg-api
            userDataSecret:
              name: master-user-data
            zone: europe-southwest1-a
      
      
      $ oc apply -f new-master0-machine.yaml  machine.machine.openshift.io/case-03507533-rfctg-master-0-v2 created [lperezbe@lperezbe auth]$ oc get machine -n openshift-machine-api NAME                                 PHASE     TYPE            REGION              ZONE                  AGE case-03507533-rfctg-master-0-v2      Running   n2-standard-8   europe-southwest1   europe-southwest1-a   9m39s
      
      cloud compute backend-services describe case-03507533-rfctg-api-internal backends: - balancingMode: CONNECTION   group: https://www.googleapis.com/compute/v1/projects/cee-gcp-emea/zones/europe-southwest1-a/instanceGroups/case-03507533-rfctg-master-europe-southwest1-a - balancingMode: CONNECTION   group: https://www.googleapis.com/compute/v1/projects/cee-gcp-emea/zones/europe-southwest1-b/instanceGroups/case-03507533-rfctg-master-europe-southwest1-b - balancingMode: CONNECTION   group: https://www.googleapis.com/compute/v1/projects/cee-gcp-emea/zones/europe-southwest1-c/instanceGroups/case-03507533-rfctg-master-europe-southwest1-c 
      
       $ gcloud compute instance-groups list
      NAME                                            LOCATION             SCOPE  NETWORK                      MANAGED  INSTANCES
      case-03507533-rfctg-master-europe-southwest1-b  europe-southwest1-b  zone   case-03507533-rfctg-network  No       1
      case-03507533-rfctg-master-europe-southwest1-a  europe-southwest1-a  zone   case-03507533-rfctg-network  No       0
      case-03507533-rfctg-master-europe-southwest1-c  europe-southwest1-c  zone   case-03507533-rfctg-network  No       1

      Steps to Reproduce:

      1.
      2.
      3.
      

      Actual results:

       

      Expected results:

       

      Additional info:

      It Could be workarrounded by manually adding the instance into the instance-group
      
      $ gcloud compute instance-groups unmanaged add-instances case-03507533-rfctg-master-europe-southwest1-a --zone=europe-southwest1-a --instances=case-03507533-rfctg-master-0-v2
      Updated [https://www.googleapis.com/compute/v1/projects/cee-gcp-emea/zones/europe-southwest1-a/instanceGroups/case-03507533-rfctg-master-europe-southwest1-a]. gcloud compute instance-groups list
      NAME                                            LOCATION             SCOPE  NETWORK                      MANAGED  INSTANCES
      case-03507533-rfctg-master-europe-southwest1-b  europe-southwest1-b  zone   case-03507533-rfctg-network  No       1
      case-03507533-rfctg-master-europe-southwest1-a  europe-southwest1-a  zone   case-03507533-rfctg-network  No       1
      case-03507533-rfctg-master-europe-southwest1-c  europe-southwest1-c  zone   case-03507533-rfctg-network  No       1
      

      Attachments

        Issue Links

          Activity

            People

              rmanak@redhat.com Radek Manak
              rhn-support-lperezbe Luis Perez Besa
              Milind Yadav Milind Yadav
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: