Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-7931

Post OCP upgrade network ClusterOperator failed due to "ovnkube-master" 'nbdb'

XMLWordPrintable

    • No
    • Rejected
    • False
    • Hide

      None

      Show
      None

      Observed the following failure logs under ovnkube-master containers while bringing up an OCP OVNKubernetes cluster:

      ovnkube-master-xwrfd 2/6 CrashLoopBackOff 8 (22s ago) 49s

      LAST SEEN TYPE REASON OBJECT MESSAGE
      110s Warning ErrorAddingLogicalPort pod/ovnkube-master-xwrfd failed to ensurePod openshift-ovn-kubernetes/ovnkube-master-xwrfd since it is not yet scheduled
      105s Warning FailedPostStartHook pod/ovnkube-master-xwrfd Exec lifecycle hook ([/bin/bash -c set -x

      echo "$(date -Iseconds) - ERROR RESTARTING - nbdb - too many failed ovn-nbctl attempts, giving up"

      ]) for Container "nbdb" in Pod "ovnkube-master-xwrfd_openshift-ovn-kubernetes(01b0013e-caee-4697-941c-ab1e781da81e)" failed - error: command '/bin/bash -c set -x

      echo "Failed to set northd probe interval to ${northd_probe_interval}. retrying....."

      I0210 02:13:49.370552 1 main.go:339] Starting TCP socket on :9102
      F0210 02:13:49.378133 1 main.go:342] failed to listen on secure address: listen tcp :9102: bind: address already in use
      goroutine 1 [running]:

      2023-02-10T02:13:49.883Z|00001|vlog|INFO|opened log file /var/log/ovn/ovsdb-server-sb.log
      2023-02-10T02:13:49.883Z|00002|lockfile|WARN|/etc/ovn/.ovnsb_db.db.lock: failed to lock file: Resource temporarily unavailable
      ovsdb-server: I/O error: /etc/ovn/ovnsb_db.db: failed to lock lockfile (Resource temporarily unavailable)

      I0210 02:12:04.408789 1 services_controller.go:57] Creating event broadcaster
      I0210 02:12:04.408940 1 services_controller.go:72] Setting up event handlers for services
      I0210 02:12:04.408978 1 services_controller.go:82] Setting up event handlers for endpoint slices
      I0210 02:12:04.409125 1 leaderelection.go:248] attempting to acquire leader lease openshift-ovn-kubernetes/ovn-kubernetes-master...
      I0210 02:12:04.417889 1 leaderelection.go:352] lock is held by master1.ocp4.vlan70.mcp and has not yet expired
      I0210 02:12:04.417938 1 leaderelection.go:253] failed to acquire lease openshift-ovn-kubernetes/ovn-kubernetes-master
      I0210 02:12:04.417952 1 master.go:116] Lost the election to master1.ocp4.vlan70.mcp; in standby mode
      2023-02-10T02:12:14.151Z|00008|memory|INFO|8388 kB peak resident set size after 10.0 seconds
      I0210 02:12:46.994609 1 leaderelection.go:352] lock is held by master1.ocp4.vlan70.mcp and has not yet expired
      I0210 02:12:46.994638 1 leaderelection.go:253] failed to acquire lease openshift-ovn-kubernetes/ovn-kubernetes-master
      I0210 02:13:22.947664 1 leaderelection.go:352] lock is held by master1.ocp4.vlan70.mcp and has not yet expired
      I0210 02:13:22.947698 1 leaderelection.go:253] failed to acquire lease openshift-ovn-kubernetes/ovn-kubernetes-master

            mcambria Michael Cambria
            rhn-support-evadla Eswar Vadla (Inactive)
            Anurag Saxena Anurag Saxena
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: