Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-34155

Panic in MAPO deleting Machine in ERROR state

XMLWordPrintable

    • ?
    • Important
    • No
    • ShiftStack Sprint 254
    • 1
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the `machine-api` Operator reacted when it deleted a server that was in an `ERROR` state. This happened because the server did not pass a port list. With this release, deleting a machine stuck in an `ERROR` state does not cause an Operator reaction. (link:https://issues.redhat.com/browse/OCPBUGS-34155[*OCPBUGS-34155*])
      Show
      * Previously, the `machine-api` Operator reacted when it deleted a server that was in an `ERROR` state. This happened because the server did not pass a port list. With this release, deleting a machine stuck in an `ERROR` state does not cause an Operator reaction. (link: https://issues.redhat.com/browse/OCPBUGS-34155 [* OCPBUGS-34155 *])
    • Bug Fix
    • Done

      This is a clone of issue OCPBUGS-33806. The following is the description of the original issue:

      I0516 19:40:24.080597       1 controller.go:156] mbooth-psi-ph2q7-worker-0-9z9nn: reconciling Machine
      I0516 19:40:24.113866       1 controller.go:200] mbooth-psi-ph2q7-worker-0-9z9nn: reconciling machine triggers delete
      I0516 19:40:32.487925       1 controller.go:115]  "msg"="Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference" "controller"="machine-controller" "name"="mbooth-psi-ph2q7-worker-0-9z9nn" "namespace"="openshift-machine-api" "object"={"name":"mbooth-psi-ph2q7-worker-0-9z9nn","namespace":"openshift-machine-api"} "reconcileID"="f477312c-dd62-49b2-ad08-28f48c506c9a"
      panic: runtime error: invalid memory address or nil pointer dereference [recovered]
              panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x242a275]
      
      goroutine 317 [running]:
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
              /go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116 +0x1e5
      panic({0x29cfb00?, 0x40f1d50?})
              /usr/lib/golang/src/runtime/panic.go:914 +0x21f
      sigs.k8s.io/cluster-api-provider-openstack/pkg/cloud/services/compute.(*Service).constructPorts(0x3056b80?, 0xc00074d3d0, 0xc0004fe100)
              /go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/cluster-api-provider-openstack/pkg/cloud/services/compute/instance.go:188 +0xb5
      sigs.k8s.io/cluster-api-provider-openstack/pkg/cloud/services/compute.(*Service).DeleteInstance(0xc00074d388, 0xc000c61300?, {0x3038ae8, 0xc0008b7440}, 0xc00097e2a0, 0xc0004fe100)
              /go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/cluster-api-provider-openstack/pkg/cloud/services/compute/instance.go:678 +0x42d
      github.com/openshift/machine-api-provider-openstack/pkg/machine.(*OpenstackClient).Delete(0xc0001f2380, {0x304f708?, 0xc000c6df80?}, 0xc0008b7440)
              /go/src/sigs.k8s.io/cluster-api-provider-openstack/pkg/machine/actuator.go:341 +0x305
      github.com/openshift/machine-api-operator/pkg/controller/machine.(*ReconcileMachine).Reconcile(0xc00045de50, {0x304f708, 0xc000c6df80}, {{{0xc00066c7f8?, 0x0?}, {0xc000dce980?, 0xc00074dd48?}}})
              /go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/github.com/openshift/machine-api-operator/pkg/controller/machine/controller.go:216 +0x1cfe
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x3052e08?, {0x304f708?, 0xc000c6df80?}, {{{0xc00066c7f8?, 0xb?}, {0xc000dce980?, 0x0?}}})
              /go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119 +0xb7
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0004eb900, {0x304f740, 0xc00045c500}, {0x2ac0340?, 0xc0001480c0?})
              /go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316 +0x3cc
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0004eb900, {0x304f740, 0xc00045c500})
              /go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 +0x1c9
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
              /go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 +0x79
      created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 269
              /go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:223 +0x565
      
      > kc get clusterversion
      NAME      VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.16.0-ec.6   True        False         7d3h    Cluster version is 4.16.0-ec.6
      
      > kc -n openshift-machine-api get machines.m mbooth-psi-ph2q7-worker-0-9z9nn -o yaml
      apiVersion: machine.openshift.io/v1beta1
      kind: Machine
      metadata:
        annotations:
          machine.openshift.io/instance-state: ERROR
          openstack-resourceId: dc08c2a2-cbda-4892-a06b-320d02ec0c6c
        creationTimestamp: "2024-05-16T16:53:16Z"
        deletionGracePeriodSeconds: 0
        deletionTimestamp: "2024-05-16T19:23:44Z"
        finalizers:
        - machine.machine.openshift.io
        generateName: mbooth-psi-ph2q7-worker-0-
        generation: 3
        labels:
          machine.openshift.io/cluster-api-cluster: mbooth-psi-ph2q7
          machine.openshift.io/cluster-api-machine-role: worker
          machine.openshift.io/cluster-api-machine-type: worker
          machine.openshift.io/cluster-api-machineset: mbooth-psi-ph2q7-worker-0
          machine.openshift.io/instance-type: ci.m1.xlarge
          machine.openshift.io/region: regionOne
          machine.openshift.io/zone: ""
        name: mbooth-psi-ph2q7-worker-0-9z9nn
        namespace: openshift-machine-api
        ownerReferences:
        - apiVersion: machine.openshift.io/v1beta1
          blockOwnerDeletion: true
          controller: true
          kind: MachineSet
          name: mbooth-psi-ph2q7-worker-0
          uid: f715dba2-b0b2-4399-9ab6-19daf6407bd7
        resourceVersion: "8391649"
        uid: 6d1ad181-5633-43eb-9b19-7c73c86045c3
      spec:
        lifecycleHooks: {}
        metadata: {}
        providerID: openstack:///dc08c2a2-cbda-4892-a06b-320d02ec0c6c
        providerSpec:
          value:
            apiVersion: machine.openshift.io/v1alpha1
            cloudName: openstack
            cloudsSecret:
              name: openstack-cloud-credentials
              namespace: openshift-machine-api
            flavor: ci.m1.xlarge
            image: ""
            kind: OpenstackProviderSpec
            metadata:
              creationTimestamp: null
            networks:
            - filter: {}
              subnets:
              - filter:
                  tags: openshiftClusterID=mbooth-psi-ph2q7
            rootVolume:
              diskSize: 50
              sourceUUID: rhcos-4.16
              volumeType: tripleo
            securityGroups:
            - filter: {}
              name: mbooth-psi-ph2q7-worker
            serverGroupName: mbooth-psi-ph2q7-worker
            serverMetadata:
              Name: mbooth-psi-ph2q7-worker
              openshiftClusterID: mbooth-psi-ph2q7
            tags:
            - openshiftClusterID=mbooth-psi-ph2q7
            trunk: true
            userDataSecret:
              name: worker-user-data
      status:
        addresses:
        - address: mbooth-psi-ph2q7-worker-0-9z9nn
          type: Hostname
        - address: mbooth-psi-ph2q7-worker-0-9z9nn
          type: InternalDNS
        conditions:
        - lastTransitionTime: "2024-05-16T16:56:05Z"
          status: "True"
          type: Drainable
        - lastTransitionTime: "2024-05-16T19:24:26Z"
          message: Node drain skipped
          status: "True"
          type: Drained
        - lastTransitionTime: "2024-05-16T17:14:59Z"
          status: "True"
          type: InstanceExists
        - lastTransitionTime: "2024-05-16T16:56:05Z"
          status: "True"
          type: Terminable
        lastUpdated: "2024-05-16T19:23:52Z"
        phase: Deleting
      

            rhn-gps-mbooth Matthew Booth
            openshift-crt-jira-prow OpenShift Prow Bot
            Itshak Brown Itshak Brown
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: