ovn-northd: northd/northd.c:5860: assertion od failed in northd_handle_lb_data_changes()
2024-04-15T10:45:43Z|00001|fatal_signal(stopwatch0)|WARN|terminating with signal 11 (Segmentation fault)

[core@crc ~]$ coredumpctl list|tail -1
Wed 2024-04-17 13:16:06 UTC 1214619 1000660000   0 SIGSEGV inaccessible /usr/bin/ovn-northd    -
[core@crc ~]$ sudo coredumpctl dump 1214619 --output 1214619.core
           PID: 1214619 (ovn-northd)
           UID: 1000660000 (1000660000)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Wed 2024-04-17 13:16:03 UTC (5min ago)
  Command Line: /usr/bin/ovn-northd -vfile:off -vconsole:info --n-threads=1 --ovnnb-db=ssl:ovsdbserver-nb-0.openstack.svc.cluster.local:6641 --ovnsb-db=ssl:ovsdbserver-sb-0.openstack.svc.cluster.local:6642 --certificate=/etc/pki/tls/certs/ovndb.crt --private-key=/etc/pki/tls/private/ovndb.key --ca-cert=/etc/pki/tls/certs/ovndbca.crt
    Executable: /usr/bin/ovn-northd
 Control Group: /kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode78b666a_c8d1_42d0_a336_6784b88af329.slice/crio-6caed909a12866e57bf38fac80c0ea2f117c930e59290d0ac9241b98aa25bf1a.scope
          Unit: crio-6caed909a12866e57bf38fac80c0ea2f117c930e59290d0ac9241b98aa25bf1a.scope
         Slice: kubepods-besteffort-pode78b666a_c8d1_42d0_a336_6784b88af329.slice
       Boot ID: 3bac4cb5d608415090b0e2b33a4911da
    Machine ID: 76a652f77ec14a008450a14489d678b8
      Hostname: ovn-northd-67b7fc57-jgk26
       Storage: /var/lib/systemd/coredump/core.ovn-northd.1000660000.3bac4cb5d608415090b0e2b33a4911da.1214619.1713359763000000.zst (present)
  Size on Disk: 770.6K
       Message: Process 1214619 (ovn-northd) of user 1000660000 dumped core.
                
                Stack trace of thread 1:
                #0  0x00007f57331cb898 n/a (/usr/lib64/libc.so.6 + 0x28898)
                ELF object binary architecture: AMD x86-64
(gdb) info threads
  Id   Target Id                     Frame 

1    Thread 0x7f5732d91a00 (LWP 1) 0x00007f57331cb898 in abort () from /lib64/libc.so.6
  2    Thread 0x7f5732574640 (LWP 8) 0x00007f5733229960 in __lll_lock_wait () from /lib64/libc.so.6
  3    Thread 0x7f5732d75640 (LWP 7) 0x00007f573320f87b in __vfprintf_internal () from /lib64/libc.so.6
(gdb) where 
#0  0x00007f57331cb898 in abort () from /lib64/libc.so.6
#1  0x000055b3ceaa6120 in ovs_abort_valist (args=0x7fff5c1366f0, format=0x55b3ceabeb38 "%s: assertion %s failed in %s()", err_no=0)
    at ovs-49e64f13b2c965f5b53a65eeab70ac2e3f0bf69a/lib/util.c:447
#2  vlog_abort_valist (module_=<optimized out>, message=0x55b3ceabeb38 "%s: assertion %s failed in %s()", args=0x7fff5c1366f0, 
    message=0x55b3ceabeb38 "%s: assertion %s failed in %s()", module_=<optimized out>)
    at ovs-49e64f13b2c965f5b53a65eeab70ac2e3f0bf69a/lib/vlog.c:1293
#3  vlog_abort.constprop.0 (module=<optimized out>, message=0x55b3ceabeb38 "%s: assertion %s failed in %s()", 
    message=0x55b3ceabeb38 "%s: assertion %s failed in %s()", module=<optimized out>)
    at ovs-49e64f13b2c965f5b53a65eeab70ac2e3f0bf69a/lib/vlog.c:1307
#4  0x000055b3cea8d32b in ovs_assert_failure (where=<optimized out>, function=<optimized out>, condition=<optimized out>)
    at ovs-49e64f13b2c965f5b53a65eeab70ac2e3f0bf69a/lib/util.c:89
#5  0x000055b3cea5003a in northd_handle_lb_data_changes (lbgrp_datapaths_map=0x55b3cf843190, lb_datapaths_map=0x55b3cf843170, 
    lr_datapaths=0x55b3cf843108, ls_datapaths=0x55b3cf8430e0, trk_lb_data=<optimized out>) at northd/northd.c:5860
#6  northd_handle_lb_data_changes (lbgrp_datapaths_map=0x55b3cf843190, lb_datapaths_map=0x55b3cf843170, lr_datapaths=0x55b3cf843108, 
    ls_datapaths=0x55b3cf8430e0, trk_lb_data=<optimized out>) at northd/northd.c:5722
#7  northd_lb_data_handler (node=<optimized out>, data=0x55b3cf8430e0) at northd/en-northd.c:229
#8  0x000055b3cea66b23 in engine_compute (recompute_allowed=<optimized out>, node=<optimized out>) at lib/inc-proc-eng.c:441
#9  engine_run_node (recompute_allowed=true, node=0x55b3ceb1cae0 <en_northd>) at lib/inc-proc-eng.c:503
#10 engine_run (recompute_allowed=true) at lib/inc-proc-eng.c:528
#11 engine_run (recompute_allowed=true) at lib/inc-proc-eng.c:517
#12 inc_proc_northd_run (ovnnb_txn=<optimized out>, ovnsb_txn=<optimized out>, ctx=0x7fff5c136a10) at northd/inc-proc-northd.c:364
#13 0x000055b3cea18ada in main (argc=<optimized out>, argv=<optimized out>) at northd/ovn-northd.c:961
 

    def test_port_forwarding_info_in_fip_details(self):
        fip = self.create_floatingip()
        port = self.create_port(self.network)
        # Ensure that FIP don't have information about any port forwarding yet
        fip = self.client.show_floatingip(fip['id'])['floatingip']
        self.assertEqual(0, len(fip['port_forwardings']))
        # Now create port forwarding and ensure that it is visible in FIP's
        # details
        pf = self.create_port_forwarding(
            fip['id'],
            internal_port_id=port['id'],
            internal_ip_address=port['fixed_ips'][0]['ip_address'],
            internal_port=1111, external_port=2222, protocol="tcp")
        fip = self.client.show_floatingip(fip['id'])['floatingip']
        self.assertEqual(1, len(fip['port_forwardings']))
        self.assertEqual(1111, fip['port_forwardings'][0]['internal_port'])
        self.assertEqual(2222, fip['port_forwardings'][0]['external_port'])
        self.assertEqual('tcp', fip['port_forwardings'][0]['protocol'])
        self.assertEqual(port['fixed_ips'][0]['ip_address'],
                         fip['port_forwardings'][0]['internal_ip_address'])
        # Delete port forwarding and ensure that it's not in FIP's details
        # anymore
        self.client.delete_port_forwarding(fip['id'], pf['id'])
        fip = self.client.show_floatingip(fip['id'])['floatingip']
        self.assertEqual(0, len(fip['port_forwardings']))