Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-895

Machine Controller stuck with Terminated Instances while Provisioning on AWS



    • Important
    • CLOUD Sprint 224
    • 1
    • False
    • Hide




      This is a clone of issue OCPBUGS-572. The following is the description of the original issue:

      This is a clone of BugĀ 2117557 to track backport to 4.9.z
      +++ This bug was initially created as a clone of
      Bug #2108021

      +++ This bug was initially created as a clone of
      Bug #2106733

      Description of problem:
      During a replacement of worker nodes, we noticed that the machine-controller container, which is deployed as part of the `openshift-machine-api` namespace, would panic when a machine OpenShift was still in "Provisioning" state, but the corresponding AWS instance was already "Terminated".

      I0628 10:09:02.518169 1 reconciler.go:123] my-super-worker-skghqwd23: deleting machine
      I0628 10:09:03.090641 1 reconciler.go:464] my-super-worker-skghqwd23: Found instance by id: i-11111111111111
      I0628 10:09:03.090662 1 reconciler.go:138] my-super-worker-skghqwd23: found 1 existing instances for machine
      I0628 10:09:03.090669 1 utils.go:231] Cleaning up extraneous instance for machine: i-11111111111111, state: running, launchTime: 2022-06-28 08:56:52 +0000 UTC
      I0628 10:09:03.090682 1 utils.go:235] Terminating i-05332b08d4cc3ab28 instance
      panic: assignment to entry in nil map

      goroutine 125 [running]:
      sigs.k8s.io/cluster-api-provider-aws/pkg/actuators/machine.(*Reconciler).delete(0xc0012df980, 0xc0004bd530, 0x234c4c0)
      /go/src/sigs.k8s.io/cluster-api-provider-aws/pkg/actuators/machine/reconciler.go:165 +0x95b
      sigs.k8s.io/cluster-api-provider-aws/pkg/actuators/machine.(*Actuator).Delete(0xc000a3a900, 0x25db9b8, 0xc0004bd530, 0xc000b9a000, 0x35e0100, 0x0)
      /go/src/sigs.k8s.io/cluster-api-provider-aws/pkg/actuators/machine/actuator.go:171 +0x365
      github.com/openshift/machine-api-operator/pkg/controller/machine.(*ReconcileMachine).Reconcile(0xc0007bc960, 0x25db9b8, 0xc0004bd530, 0xc0007c5fc8, 0x15, 0xc0005e4a80, 0x2a, 0xc0004bd530, 0xc000032000, 0x206d640, ...)
      /go/src/sigs.k8s.io/cluster-api-provider-aws/vendor/github.com/openshift/machine-api-operator/pkg/controller/machine/controller.go:231 +0x2352
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0003b20a0, 0x25db910, 0xc00087e040, 0x1feb8e0, 0xc00009f460)
      /go/src/sigs.k8s.io/cluster-api-provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298 +0x30d
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0003b20a0, 0x25db910, 0xc00087e040, 0x0)
      /go/src/sigs.k8s.io/cluster-api-provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253 +0x205
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2(0xc000a38790, 0xc0003b20a0, 0x25db910, 0xc00087e040)
      /go/src/sigs.k8s.io/cluster-api-provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214 +0x6b
      created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
      /go/src/sigs.k8s.io/cluster-api-provider-aws/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:210 +0x425

      What is the business impact? Please also provide timeframe information.
      We failed to recover from a major outage due to this bug.

      Where are you experiencing the behavior? What environment?
      Production and all envs.

      When does the behavior occur? Frequency? Repeatedly? At certain times?
      It appeared only once so far, but can appear in larger scaling scenarios.

      Version-Release number of selected component (if applicable):

      Actual results:

      With the panicing machine-controller, no new instances could be provisioned, resulting in an unscalable cluster. The solution/workaround to the problem was to delete the offending Machines.
      Expected results:
      Make the cluster scaleable again without deleting manually.

      Additional info:

      — Additional comment from
      on 2022-07-13 13:34:08 UTC —

      Probably the issue is here:

      Since issue is in machine-api, moving it to correct team.

      — Additional comment from
      on 2022-07-14 08:20:00 UTC —

      I am working on a fix for this.

      — Additional comment from
      on 2022-07-14 19:10:14 UTC —

      Elliott changed bug status from MODIFIED to ON_QA.
      This bug is expected to ship in the next 4.12 release.

      — Additional comment from
      on 2022-07-18 15:18:16 UTC —

      Waiting for the first 4.11.z stream before we merge

      — Additional comment from
      on 2022-08-08 15:07:16 UTC —

      Waiting on 4.11 GA to move ahead here


        Issue Links



              rmanak@redhat.com Radek Manak
              openshift-crt-jira-prow OpenShift Prow Bot
              Huali Liu Huali Liu
              0 Vote for this issue
              5 Start watching this issue