Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-32717

During rolling back to 4.15.0 from 4.15.0 nightly build, one ovnkube-node pod got CrashLoopBackOff

XMLWordPrintable

    • Low
    • No
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      During rolling back to 4.15.0 from 4.15.0 nightly build, one ovnkube-node pod got CrashLoopBackOff

      Version-Release number of selected component (if applicable):

         

      How reproducible:

          Always

      Steps to Reproduce:

      Run prow ci job: 
      https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-rollback-nightly-azure-upi-f14/1781132193834209280
      
      omc describe pod -n openshift-ovn-kubernetes ovnkube-node-h9lv5
      …
      Last State:  Terminated
            Reason:    Error
            Message:   es/go-controller/pkg/crd/egressservice/v1/apis/informers/externalversions/factory.go:140
      I0419 06:00:14.344338  109993 reflector.go:295] Stopping reflector *v1.Pod (0s) from k8s.io/client-go/informers/factory.go:159
      I0419 06:00:14.344422  109993 reflector.go:295] Stopping reflector *v1.EgressQoS (0s) from github.com/openshift/ovn-kubernetes/go-controller/pkg/crd/egressqos/v1/apis/informers/externalversions/factory.go:140
      I0419 06:00:14.344524  109993 reflector.go:295] Stopping reflector *v1.EgressFirewall (0s) from github.com/openshift/ovn-kubernetes/go-controller/pkg/crd/egressfirewall/v1/apis/informers/externalversions/factory.go:140
      I0419 06:00:14.344666  109993 reflector.go:295] Stopping reflector *v1.EgressIP (0s) from github.com/openshift/ovn-kubernetes/go-controller/pkg/crd/egressip/v1/apis/informers/externalversions/factory.go:140
      I0419 06:00:14.344769  109993 reflector.go:295] Stopping reflector *v1.Service (0s) from k8s.io/client-go/informers/factory.go:159
      I0419 06:00:14.344824  109993 reflector.go:295] Stopping reflector *v1.Namespace (0s) from k8s.io/client-go/informers/factory.go:159
      I0419 06:00:14.344786  109993 reflector.go:295] Stopping reflector *v1.Node (0s) from k8s.io/client-go/informers/
            Exit Code:    1
      
      Must-gather: 
      https://gcsweb-qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-rollback-nightly-azure-upi-f14/1781132193834209280/artifacts/azure-upi-f14/gather-must-gather/artifacts/ 
      
      Run another prow ci job:
      https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-rollback-nightly-gcp-ipi-ovn-ipsec-f14/1781805900550377472
      
      omc describe pods -n openshift-ovn-kubernetes ovnkube-node-kjctb
      …
      State:       Waiting
            Reason:    CrashLoopBackOff
          Last State:  Terminated
            Reason:    Error
            Message:   ressip/v1/apis/informers/externalversions/factory.go:140
      I0421 02:59:19.996673  171025 reflector.go:295] Stopping reflector *v1.Pod (0s) from k8s.io/client-go/informers/factory.go:159
      I0421 02:59:19.996720  171025 reflector.go:295] Stopping reflector *v1.Node (0s) from k8s.io/client-go/informers/factory.go:159
      I0421 02:59:19.996884  171025 reflector.go:295] Stopping reflector *v1.EgressService (0s) from github.com/openshift/ovn-kubernetes/go-controller/pkg/crd/egressservice/v1/apis/informers/externalversions/factory.go:140
      I0421 02:59:19.996804  171025 reflector.go:295] Stopping reflector *v1.NetworkPolicy (0s) from k8s.io/client-go/informers/factory.go:159
      I0421 02:59:19.996846  171025 metrics.go:506] Stopping metrics server 127.0.0.1:29103
      I0421 02:59:19.996941  171025 reflector.go:295] Stopping reflector *v1.Service (0s) from k8s.io/client-go/informers/factory.go:159
      I0421 02:59:19.996976  171025 metrics.go:502] Metrics server has stopped serving at address "127.0.0.1:29103"
      I0421 02:59:19.996995  171025 reflector.go:295] Stopping reflector *v1.EgressQoS (0s) from github.com/openshift/ovn-kubernetes/go-controller/pkg/crd/egressqos/v1/apis/informers/externalversions/factory.go:140
      I0421 02:59:19.996865  171025 reflector.go:295] Stopping reflector *v1.AdminPolicyBasedExternalRoute (0s) from github.com/openshift/ovn-kubernetes/go-controller/pk
            Exit Code:    1
      

      Actual results:

      One ovnkube-node pod got CrashLoopBackOff during rollback    

      Expected results:

      Rollback should be successful    

      Additional info:

          

            sdn-team-bot sdn-team bot
            minl@redhat.com Ming Li
            Anurag Saxena Anurag Saxena
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: