Uploaded image for project: 'Fast Datapath Product'
  1. Fast Datapath Product
  2. FDP-2786

QE verification: [ovn-northd] assertion failure due to trying to write to a deleted IDL record

    • Icon: Task Task
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • None
    • ovn25.09
    • False
    • Hide

      None

      Show
      None
    • False
    • Hide

      ( ) The bug has been reproduced and verified by QE members
      ( ) Test coverage has been added to downstream CI
      ( ) For new feature, failed test plans have bugs added as children to the epic
      ( ) The bug is cloned to any relevant release that we support and/or is needed

      Show
      ( ) The bug has been reproduced and verified by QE members ( ) Test coverage has been added to downstream CI ( ) For new feature, failed test plans have bugs added as children to the epic ( ) The bug is cloned to any relevant release that we support and/or is needed
    • rhel-9
    • None
    • OVN FDP Sprint 13
    • 1

      This ticket is tracking the QE verification effort for the solution to the problem described below.

       Problem Description: Clearly explain the issue.

      With ovn25.09-25.09.1-11.el9fdp ovn-northd hits an assertion failure:

      2025-12-01T18:33:16.043Z|00021|backtrace|ERR|lib/vlog.c:1309 backtrace:
      ovn-northd(+0xd3d37) [0x5616a6f33d37]
      ovn-northd(+0xc07c3) [0x5616a6f207c3]
      ovn-northd(+0xb487b) [0x5616a6f1487b]
      ovn-northd(+0xd50fa) [0x5616a6f350fa]
      ovn-northd(+0x77442) [0x5616a6ed7442]
      ovn-northd(+0x85397) [0x5616a6ee5397]
      ovn-northd(+0x87a8a) [0x5616a6ee7a8a]
      ovn-northd(+0x2012f) [0x5616a6e8012f]
      /lib64/libc.so.6(+0x29590) [0x7f6b92352590]
      /lib64/libc.so.6(__libc_start_main+0x80) [0x7f6b92352640]
      ovn-northd(+0x21035) [0x5616a6e81035]
      ovn-northd: lib/ovsdb-idl.c:3650: assertion row->new_datum != NULL failed in ovsdb_idl_txn_write__()
      

       
      Decoded backtrace:

      ovn-northd(+0xb487b) [0x5616a6f1487b]    ovs_assert_failure (ovs-2e7d3ea0d0cade43da9156646cf8260f454431e8/lib/util.c:88).
      ovn-northd(+0xd50fa) [0x5616a6f350fa]    ovsdb_idl_txn_write__()
      ovn-northd(+0x77442) [0x5616a6ed7442]    en_sync_from_sb_run()
      ovn-northd(+0x85397) [0x5616a6ee5397]
      ovn-northd(+0x87a8a) [0x5616a6ee7a8a]
      
      
      
      (gdb) list *0x77442
      0x77442 is in en_sync_from_sb_run (northd/northd.c:19564).
      19559           }
      19560
      19561           /* ovn-controller will update 'Port_Binding.up' only if it was
      19562            * explicitly set to 'false'.
      19563            */
      19564           if (!op->sb->n_up) {
      19565               up = false;
      19566               sbrec_port_binding_set_up(op->sb, &up, 1);
      19567           }
      

      Hit here: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshi[...]bernetes_ovnkube-node-sxwmc_northd_previous.log

       Impact Assessment: Describe the severity and impact (e.g., network down,availability of a workaround, etc.).

      northd crashes
       

       Software Versions: Specify the exact versions in use (e.g.,openvswitch3.1-3.1.0-147.el8fdp).

      ovn25.09-25.09.1-11.el9fdp
       

        Issue Type: Indicate whether this is a new issue or a regression (if a regression, state the last known working version).

      unknown
       

       Reproducibility: Confirm if the issue can be reproduced consistently. If not, describe how often it occurs.

      Sometimes in OCP CI, e.g.:
      https://github.com/openshift/ovn-kubernetes/pull/2884
       

       Reproduction Steps: Provide detailed steps or scripts to replicate the issue.

      Unknown.
       

       Expected Behavior: Describe what should happen under normal circumstances.

      ovn-northd should not crash
       

       Observed Behavior: Explain what actually happens.

      ovn-northd tries to write to a deleted IDL record triggering an assertion failure in the IDL.
       

       Troubleshooting Actions: Outline the steps taken to diagnose or resolve the issue so far.

       

       Logs: If you collected logs please provide them (e.g. sos report, /var/log/openvswitch/* , testpmd console)

      See attached NB/SB databases. NOTE: they're taken way after the fact so we need to try to correlate what happened around the time of the crash (2025-12-01T18:33:16.043Z).

              ovnteam@redhat.com OVN Team
              nstbot NST Bot
              OVN QE OVN QE
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: