Uploaded image for project: 'Fast Datapath Product'
  1. Fast Datapath Product
  2. FDP-946

[ovn24.03]ovn-controller would crash with basic ovn setup

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • None
    • None
    • ovn23.09
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • rhel-sst-network-fastdatapath
    • ssg_networking
    • Critical

      description:

      with simple ovn setup, ovn-controller would crash

      version:

      ovn24.03-24.03.4-4.el9fdp.x86_64

      steps:

      systemctl start openvswitch                          
      systemctl start ovn-northd
      ovn-nbctl set-connection ptcp:6641
      ovn-sbctl set-connection ptcp:6642
      ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.88.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.88.25
      systemctl restart ovn-controller
      ovs-vsctl add-br br-ext
      ovs-vsctl set Open_vSwitch . external-ids:ovn-bridge-mappings=phynet:br-ext
      ovs-vsctl add-port br-ext ens1f1np1
      ip link set ens1f1np1 up
      ovn-nbctl lr-add lr1
      ovn-nbctl lrp-add lr1 lr1-ls1 00:00:01:ff:02:03 192.168.1.254/24 1111::a/64
      ovn-nbctl ls-add ls1
      ovn-nbctl lsp-add ls1 ls1p1
      ovn-nbctl lsp-set-addresses ls1p1 "00:00:01:01:01:01 192.168.1.1 1111::1"
      ovn-nbctl lsp-add ls1 ls1p2
      ovn-nbctl lsp-set-addresses ls1p2 "00:00:01:01:01:02 192.168.1.12 1111::2"
      ovn-nbctl lsp-add ls1 ls1-lr1
      ovn-nbctl lsp-set-type ls1-lr1 router
      ovn-nbctl lsp-set-options ls1-lr1 router-port=lr1-ls1
      ovn-nbctl lsp-set-addresses ls1-lr1 router
      ovn-nbctl ls-add ls2
      ovn-nbctl lsp-add ls2 ls2p1
      ovn-nbctl lsp-set-addresses ls2p1 "00:00:01:01:02:01 192.168.2.1 1112::1"
      ovn-nbctl lsp-add ls2 ls2p2
      ovn-nbctl lsp-set-addresses ls2p2 "00:00:01:01:02:02 192.168.2.2 1112::2"
      ovn-nbctl lrp-add lr1 lr1-ls2 00:00:01:ff:22:03 192.168.2.254/24 1112::a/64
      ovn-nbctl lsp-add ls2 ls2-lr1
      ovn-nbctl lsp-set-type ls2-lr1 router
      ovn-nbctl lsp-set-options ls2-lr1 router-port=lr1-ls2
      ovn-nbctl lsp-set-addresses ls2-lr1 router
      ovn-nbctl ls-add pub
      ovn-nbctl lrp-add lr1 lr1-pub 00:00:01:ff:01:03 172.16.1.254/24 172:16::a/64
      ovn-nbctl lrp-set-gateway-chassis lr1-pub hv1
      ovn-nbctl lsp-add pub pub-lr1
      ovn-nbctl lsp-set-type pub-lr1 router
      ovn-nbctl lsp-set-addresses pub-lr1 router
      ovn-nbctl lsp-set-options pub-lr1 router-port=lr1-pub
      ovn-nbctl lsp-add pub pub-ln
      ovn-nbctl lsp-set-type pub-ln localnet
      ovn-nbctl lsp-set-addresses pub-ln unknown
      ovn-nbctl lsp-set-options pub-ln network_name=phynet
      ovn-nbctl lsp-add ls1 ls1-ln
      ovn-nbctl lsp-set-type ls1-ln localnet
      ovn-nbctl lsp-set-addresses ls1-ln unknown
      ovn-nbctl lsp-set-options ls1-ln network_name=phynet
      ovn-nbctl lsp-add ls2 ls2-ln
      ovn-nbctl lsp-set-type ls2-ln localnet
      ovn-nbctl lsp-set-addresses ls2-ln unknown
      ovn-nbctl lsp-set-options ls2-ln network_name=phynet
      ovn-nbctl set logical_switch_port ls2-ln tag_request=50
      ovn-nbctl lr-nat-add lr1 dnat_and_snat 172.16.1.21 192.168.2.1 ls2p1 00:00:0f:01:02:01
      ovn-nbctl lr-nat-add lr1 dnat_and_snat 172.16.1.22 192.168.2.2 ls2p2 00:00:0f:01:02:02
      ovs-vsctl add-port br-int ls1p1 -- set interface ls1p1 type=internal external_ids:iface-id=ls1p1
      ip netns add ls1p1
      ip link set ls1p1 netns ls1p1
      ip netns exec ls1p1 ip link set ls1p1 address 00:00:01:01:01:01
      ip netns exec ls1p1 ip link set ls1p1 up
      ip netns exec ls1p1 ip addr add 192.168.1.1/24 dev ls1p1
      ip netns exec ls1p1 ip route add default via 192.168.1.254
      ip netns exec ls1p1 ip addr add 1111::1/64 dev ls1p1
      ip netns exec ls1p1 ip -6 route add default via 1111::a
      ovs-vsctl add-port br-int ls2p1 -- set interface ls2p1 type=internal external_ids:iface-id=ls2p1
      ip netns add ls2p1
      ip link set ls2p1 netns ls2p1
      ip netns exec ls2p1 ip link set ls2p1 address 00:00:01:01:02:01
      ip netns exec ls2p1 ip link set ls2p1 up
      ip netns exec ls2p1 ip addr add 192.168.2.1/24 dev ls2p1
      ip netns exec ls2p1 ip route add default via 192.168.2.254
      ip netns exec ls2p1 ip addr add 1112::1/64 dev ls2p1
      ip netns exec ls2p1 ip -6 route add default via 1112::a
      ovs-vsctl add-port br-ext ext1 -- set interface ext1 type=internal
      ip netns add ext1
      ip link set ext1 netns ext1
      ip netns exec ext1 ip link set lo up
      ip netns exec ext1 ip link set ext1 up
      ip netns exec ext1 ip addr add 172.16.1.11/24 dev ext1
      ip netns exec ext1 ip addr add 172:16::11/64 dev ext1 

      actual result:

      ovn-controller crash:

      [root@wsfd-advnetlab20 test]# systemctl status ovn-controller
      × ovn-controller.service - OVN controller daemon
           Loaded: loaded (/usr/lib/systemd/system/ovn-controller.service; disabled; preset: disabled)
           Active: failed (Result: signal) since Thu 2024-10-31 02:25:27 EDT; 3min 22s ago
         Duration: 40ms
          Process: 57340 ExecStart=/usr/share/ovn/scripts/ovn-ctl --no-monitor --ovn-user=${OVN_USER_ID} start_controller $OVN_CONTROLLER_OPTS (code=exited, status=0/SUCCESS)
         Main PID: 57367 (code=killed, signal=ABRT)
              CPU: 145ms
      
      
      Oct 31 02:25:27 wsfd-advnetlab20.anl.eng.rdu2.dc.redhat.com systemd[1]: ovn-controller.service: Scheduled restart job, restart counter is at 5.
      Oct 31 02:25:27 wsfd-advnetlab20.anl.eng.rdu2.dc.redhat.com systemd[1]: Stopped OVN controller daemon.
      Oct 31 02:25:27 wsfd-advnetlab20.anl.eng.rdu2.dc.redhat.com systemd[1]: ovn-controller.service: Start request repeated too quickly.
      Oct 31 02:25:27 wsfd-advnetlab20.anl.eng.rdu2.dc.redhat.com systemd[1]: ovn-controller.service: Failed with result 'signal'.
      Oct 31 02:25:27 wsfd-advnetlab20.anl.eng.rdu2.dc.redhat.com systemd[1]: Failed to start OVN controller daemon.
      [root@wsfd-advnetlab20 test]# journalctl -xe -u ovn-controller --no-page
      Oct 31 02:25:24 wsfd-advnetlab20.anl.eng.rdu2.dc.redhat.com systemd[1]: Starting OVN controller daemon...
      ░░ Subject: A start job for unit ovn-controller.service has begun execution
      ░░ Defined-By: systemd
      ░░ Support: https://access.redhat.com/support
      ░░ 
      ░░ A start job for unit ovn-controller.service has begun execution.
      ░░ 
      ░░ The job identifier is 10782.
      Oct 31 02:25:24 wsfd-advnetlab20.anl.eng.rdu2.dc.redhat.com ovn-ctl[57058]: Starting ovn-controller.
      Oct 31 02:25:24 wsfd-advnetlab20.anl.eng.rdu2.dc.redhat.com systemd[1]: Started OVN controller daemon.
      ░░ Subject: A start job for unit ovn-controller.service has finished successfully
      ░░ Defined-By: systemd
      ░░ Support: https://access.redhat.com/support
      ░░ 
      ░░ A start job for unit ovn-controller.service has finished successfully.
      ░░ 
      ░░ The job identifier is 10782.
      Oct 31 02:25:25 wsfd-advnetlab20.anl.eng.rdu2.dc.redhat.com systemd[1]: ovn-controller.service: Main process exited, code=killed, status=6/ABRT
      ░░ Subject: Unit process exited
      ░░ Defined-By: systemd
      ░░ Support: https://access.redhat.com/support
      ░░ 
      ░░ An ExecStart= process belonging to unit ovn-controller.service has exited.
      ░░ 
      ░░ The process' exit code is 'killed' and its exit status is 6.
      Oct 31 02:25:25 wsfd-advnetlab20.anl.eng.rdu2.dc.redhat.com systemd[1]: ovn-controller.service: Failed with result 'signal'.
      ░░ Subject: Unit failed
      ░░ Defined-By: systemd
      ░░ Support: https://access.redhat.com/support 

      expected result:

      ovn-controller doesn't crash

       

      the issue didn't happen on the early release ovn23.09-23.09.4-38.el9fdp

              xsimonar@redhat.com Xavier Simonart
              rhn-support-jishi Jianlin Shi
              Jianlin Shi Jianlin Shi
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: