Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-3018

panic in WaitForBootstrapComplete

XMLWordPrintable

    • Important
    • None
    • Agent Sprint 226, Agent Sprint 227
    • 2
    • Proposed
    • False
    • Hide

      None

      Show
      None
    • Release Note Not Required

      Description of problem:

      When running an overnight run in dev-scripts (COMPACT_IPV4) with repeated installs I saw this panic in WaitForBootstrapComplete occur once.
      
      level=debug msg=Agent Rest API Initialized
      E1101 05:19:09.733309 1802865 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
      goroutine 1 [running]:
      k8s.io/apimachinery/pkg/util/runtime.logPanic({0x4086520?, 0x1d875810})
          /home/stack/go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00056fb00?})
          /home/stack/go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
      panic({0x4086520, 0x1d875810})
          /usr/local/go/src/runtime/panic.go:838 +0x207
      github.com/openshift/installer/pkg/agent.(*NodeZeroRestClient).getClusterID(0xc0001341e0)
          /home/stack/go/src/github.com/openshift/installer/pkg/agent/rest.go:121 +0x53
      github.com/openshift/installer/pkg/agent.(*Cluster).IsBootstrapComplete(0xc000134190)
          /home/stack/go/src/github.com/openshift/installer/pkg/agent/cluster.go:183 +0x4fc
      github.com/openshift/installer/pkg/agent.WaitForBootstrapComplete.func1()
          /home/stack/go/src/github.com/openshift/installer/pkg/agent/waitfor.go:31 +0x77
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x1d8fa901?)
          /home/stack/go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:157 +0x3e
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0001958c0?, {0x1a53c7a0, 0xc0011d4a50}, 0x1, 0xc0001958c0)
          /home/stack/go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:158 +0xb6
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0009ab860?, 0x77359400, 0x0, 0xa?, 0x8?)
          /home/stack/go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:135 +0x89
      k8s.io/apimachinery/pkg/util/wait.Until(...)
          /home/stack/go/src/github.com/openshift/installer/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:92
      github.com/openshift/installer/pkg/agent.WaitForBootstrapComplete({0x7ffd7fccb4e3?, 0x40d7e7?})
          /home/stack/go/src/github.com/openshift/installer/pkg/agent/waitfor.go:30 +0x1bc
      github.com/openshift/installer/pkg/agent.WaitForInstallComplete({0x7ffd7fccb4e3?, 0x5?})
          /home/stack/go/src/github.com/openshift/installer/pkg/agent/waitfor.go:73 +0x56
      github.com/openshift/installer/cmd/openshift-install/agent.newWaitForInstallCompleteCmd.func1(0xc0003b6c80?, {0xc0004d67c0?, 0x2?, 0x2?})
          /home/stack/go/src/github.com/openshift/installer/cmd/openshift-install/agent/waitfor.go:73 +0x126
      github.com/spf13/cobra.(*Command).execute(0xc0003b6c80, {0xc0004d6780, 0x2, 0x2})
          /home/stack/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:876 +0x67b
      github.com/spf13/cobra.(*Command).ExecuteC(0xc0013b0a00)
          /home/stack/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:990 +0x3b4
      github.com/spf13/cobra.(*Command).Execute(...)
          /home/stack/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:918
      main.installerMain()
          /home/stack/go/src/github.com/openshift/installer/cmd/openshift-install/main.go:61 +0x2b0
      main.main()
          /home/stack/go/src/github.com/openshift/installer/cmd/openshift-install/main.go:38 +0xff
      panic: runtime error: invalid memory address or nil pointer dereference [recovered]
          panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x33d3cd3]

      Version-Release number of selected component (if applicable):

      4.12.0-0.nightly-2022-10-25-210451

      How reproducible:

      Occurred on the 12th run, all previous installs were successfule

      Steps to Reproduce:

      1.Set up dev-scripts for  AGENT_E2E_TEST_SCENARIO=COMPACT_IPV4, no mirroring
      2. Run 'make clean; make agent' in a loop
      3. After repeated installs got the failure
      

      Actual results:

      Panic in WaitForBootstrapComplete

      Expected results:

      No failure

      Additional info:

      It looks like clusterResult is used here even on failure, which causes the dereference - https://github.com/openshift/installer/blob/master/pkg/agent/rest.go#L121

       

              bfournie@redhat.com Robert Fournier
              bfournie@redhat.com Robert Fournier
              zhenying niu zhenying niu
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: