k8s.io/apimachinery/pkg/util/runtime.logPanic when deleting a namespace containing a network-attach-definition

4.13.0-0.nightly-arm64-2023-02-11-212333

50%

1.Create a project called test
2.Under test project create a network-attach-definition
3.network-attach-definition is configure.
4.Delete project test then check the state of network operator 

project.project.openshift.io "ocp-46387" deleted
STEP: Check NetworkOperatorStatus after deleting namespace 02/14/23 10:32:36.556
Feb 14 10:32:46.559: INFO: Running 'oc --kubeconfig=/home/weliang/Vscode/Config/kubeconfig1 get co network'
Feb 14 10:32:46.791: INFO: Network operator state is:NAME      VERSION                                    AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
network   4.13.0-0.nightly-arm64-2023-02-11-212333   True        False         False      63m
Feb 14 10:32:56.559: INFO: Running 'oc --kubeconfig=/home/weliang/Vscode/Config/kubeconfig1 get co network'
Feb 14 10:32:56.794: INFO: Network operator state is:NAME      VERSION                                    AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
network   4.13.0-0.nightly-arm64-2023-02-11-212333   True        False         False      63m
Feb 14 10:33:06.558: INFO: Running 'oc --kubeconfig=/home/weliang/Vscode/Config/kubeconfig1 get co network'
Feb 14 10:33:06.780: INFO: Network operator state is:NAME      VERSION                                    AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
network   4.13.0-0.nightly-arm64-2023-02-11-212333   True        False         True       63m     Error while updating operator configuration: could not apply (k8s.cni.cncf.io/v1, Kind=NetworkAttachmentDefinition) ocp-46387/secondary; Namespace error for networkattachment definition, consider possible solutions: (1) Edit config files to include existing namespace (2) Create non-existent namespace (3) Delete erroneous network-attachment-definition: could not apply (k8s.cni.cncf.io/v1, Kind=NetworkAttachmentDefinition) ocp-46387/secondary: failed to apply / update (k8s.cni.cncf.io/v1, Kind=NetworkAttachmentDefinition) ocp-46387/secondary: namespaces "ocp-46387" not found
E0214 10:33:06.780662   25962 runtime.go:79] Observed a panic: types.GinkgoError{Heading:"Your Test Panicked", Message:"When you, or your assertion library, calls Ginkgo's Fail(),\nGinkgo panics to prevent subsequent assertions from running.\n\nNormally Ginkgo rescues this panic so you shouldn't see it.\n\nHowever, if you make an assertion in a goroutine, Ginkgo can't capture the panic.\nTo circumvent this, you should call\n\n\tdefer GinkgoRecover()\n\nat the top of the goroutine that caused this panic.\n\nAlternatively, you may have made an assertion outside of a Ginkgo\nleaf node (e.g. in a container node or some out-of-band function) - please move your assertion to\nan appropriate Ginkgo node (e.g. a BeforeSuite, BeforeEach, It, etc...).", DocLink:"mental-model-how-ginkgo-handles-failure", CodeLocation:types.CodeLocation{FileName:"/home/weliang/go/src/multus/Update-for-arm-payload/openshift-tests-private/test/extended/networking/utils.go", LineNumber:896, FullStackTrace:"github.com/openshift/openshift-tests-private/test/extended/networking.checkNetworkOperatorState.func1()\n\t/home/weliang/go/src/multus/Update-for-arm-payload/openshift-tests-private/test/extended/networking/utils.go:896 +0x378\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1({0x2c5edb1, 0x0})\n\t/home/weliang/go/pkg/mod/github.com/openshift/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20221020223141-4bd0702b5791/pkg/util/wait/wait.go:222 +0x1b\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0xa0f2688?, 0xc00007e250?}, 0x2a28e3f?)\n\t/home/weliang/go/pkg/mod/github.com/openshift/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20221020223141-4bd0702b5791/pkg/util/wait/wait.go:235 +0x57\nk8s.io/apimachinery/pkg/util/wait.WaitForWithContext({0xa0f2688, 0xc00007e250}, 0xc000e25278, 0x37f986a?)\n\t/home/weliang/go/pkg/mod/github.com/openshift/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20221020223141-4bd0702b5791/pkg/util/wait/wait.go:662 +0x10c\nk8s.io/apimachinery/pkg/util/wait.poll({0xa0f2688, 0xc00007e250}, 0xd0?, 0x37f8405?, 0x10?)\n\t/home/weliang/go/pkg/mod/github.com/openshift/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20221020223141-4bd0702b5791/pkg/util/wait/wait.go:596 +0x9a\nk8s.io/apimachinery/pkg/util/wait.PollWithContext({0xa0f2688, 0xc00007e250}, 0xc0013c6200?, 0xc002f75a20?, 0x2a29187?)\n\t/home/weliang/go/pkg/mod/github.com/openshift/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20221020223141-4bd0702b5791/pkg/util/wait/wait.go:460 +0x47\nk8s.io/apimachinery/pkg/util/wait.Poll(0x8686120?, 0xa08d690?, 0x8060b40?)\n\t/home/weliang/go/pkg/mod/github.com/openshift/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20221020223141-4bd0702b5791/pkg/util/wait/wait.go:445 +0x50\ngithub.com/openshift/openshift-tests-private/test/extended/networking.checkNetworkOperatorState(0xc001200000, 0xa, 0x3c)\n\t/home/weliang/go/src/multus/Update-for-arm-payload/openshift-tests-private/test/extended/networking/utils.go:888 +0x85\ngithub.com/openshift/openshift-tests-private/test/extended/networking.glob..func16.1()\n\t/home/weliang/go/src/multus/Update-for-arm-payload/openshift-tests-private/test/extended/networking/multus.go:47 +0xa79\ngithub.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func2()\n\t/home/weliang/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.0.0-20221018202518-d6fe52fe5a42/internal/suite.go:612 +0x8d\ncreated by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode\n\t/home/weliang/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.0.0-20221018202518-d6fe52fe5a42/internal/suite.go:600 +0x60c", CustomMessage:""}} (Your Test Panicked
o.Expect(matched).To(o.BeTrue())
/home/weliang/go/src/multus/Update-for-arm-payload/openshift-tests-private/test/extended/networking/utils.go:896
  When you, or your assertion library, calls Ginkgo's Fail(),
  Ginkgo panics to prevent subsequent assertions from running.  Normally Ginkgo rescues this panic so you shouldn't see it.  However, if you make an assertion in a goroutine, Ginkgo can't capture the
  panic.
  To circumvent this, you should call      defer GinkgoRecover()  at the top of the goroutine that caused this panic.  Alternatively, you may have made an assertion outside of a Ginkgo
  leaf node (e.g. in a container node or some out-of-band function) - please
  move your assertion to
  an appropriate Ginkgo node (e.g. a BeforeSuite, BeforeEach, It, etc...).  Learn more at:
  http://onsi.github.io/ginkgo/#mental-model-how-ginkgo-handles-failure
)
goroutine 207 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x8c9b220?, 0xc0003da380})
    /home/weliang/go/pkg/mod/github.com/openshift/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20221020223141-4bd0702b5791/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0003da380?})
    /home/weliang/go/pkg/mod/github.com/openshift/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20221020223141-4bd0702b5791/pkg/util/runtime/runtime.go:49 +0x75
panic({0x8c9b220, 0xc0003da380})
    /usr/local/go/src/runtime/panic.go:884 +0x212
github.com/onsi/ginkgo/v2.Fail({0xc00311a240, 0x25}, {0xc00313e260?, 0xc00311a240?, 0xa0a72a0?})
    /home/weliang/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.0.0-20221018202518-d6fe52fe5a42/core_dsl.go:335 +0x225
github.com/onsi/gomega/internal.(*Assertion).match(0xc001b20540, {0xa0da3c0, 0xe33aec8}, 0x1, {0x0, 0x0, 0x0})
    /home/weliang/go/pkg/mod/github.com/onsi/gomega@v1.20.1/internal/assertion.go:105 +0x1f0
github.com/onsi/gomega/internal.(*Assertion).To(0xc001b20540, {0xa0da3c0, 0xe33aec8}, {0x0, 0x0, 0x0})
    /home/weliang/go/pkg/mod/github.com/onsi/gomega@v1.20.1/internal/assertion.go:61 +0xb5
github.com/openshift/openshift-tests-private/test/extended/networking.checkNetworkOperatorState.func1()
    /home/weliang/go/src/multus/Update-for-arm-payload/openshift-tests-private/test/extended/networking/utils.go:896 +0x378
k8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1({0x2c5edb1, 0x0})
    /home/weliang/go/pkg/mod/github.com/openshift/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20221020223141-4bd0702b5791/pkg/util/wait/wait.go:222 +0x1b
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0xa0f2688?, 0xc00007e250?}, 0x2a28e3f?)
    /home/weliang/go/pkg/mod/github.com/openshift/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20221020223141-4bd0702b5791/pkg/util/wait/wait.go:235 +0x57
k8s.io/apimachinery/pkg/util/wait.WaitForWithContext({0xa0f2688, 0xc00007e250}, 0xc000e25278, 0x37f986a?)
    /home/weliang/go/pkg/mod/github.com/openshift/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20221020223141-4bd0702b5791/pkg/util/wait/wait.go:662 +0x10c
k8s.io/apimachinery/pkg/util/wait.poll({0xa0f2688, 0xc00007e250}, 0xd0?, 0x37f8405?, 0x10?)
    /home/weliang/go/pkg/mod/github.com/openshift/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20221020223141-4bd0702b5791/pkg/util/wait/wait.go:596 +0x9a
k8s.io/apimachinery/pkg/util/wait.PollWithContext({0xa0f2688, 0xc00007e250}, 0xc0013c6200?, 0xc002f75a20?, 0x2a29187?)
    /home/weliang/go/pkg/mod/github.com/openshift/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20221020223141-4bd0702b5791/pkg/util/wait/wait.go:460 +0x47
k8s.io/apimachinery/pkg/util/wait.Poll(0x8686120?, 0xa08d690?, 0x8060b40?)
    /home/weliang/go/pkg/mod/github.com/openshift/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20221020223141-4bd0702b5791/pkg/util/wait/wait.go:445 +0x50
github.com/openshift/openshift-tests-private/test/extended/networking.checkNetworkOperatorState(0xc001200000, 0xa, 0x3c)
    /home/weliang/go/src/multus/Update-for-arm-payload/openshift-tests-private/test/extended/networking/utils.go:888 +0x85
github.com/openshift/openshift-tests-private/test/extended/networking.glob..func16.1()
    /home/weliang/go/src/multus/Update-for-arm-payload/openshift-tests-private/test/extended/networking/multus.go:47 +0xa79
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func2()
    /home/weliang/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.0.0-20221018202518-d6fe52fe5a42/internal/suite.go:612 +0x8d
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
    /home/weliang/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.0.0-20221018202518-d6fe52fe5a42/internal/suite.go:600 +0x60c

Test should pass as it does in 4.12.

Test case: https://github.com/openshift/openshift-tests-private/blob/master/test/extended/networking/multus.go#L15-L48
Testing log: https://privatebin.corp.redhat.com/?0a95600dcde819c5#Ax5w2L5rLF7fHL58EMKu7EuhDqzBvZQTSHPGEUd4Dpp7