Description of problem:

IPSEC enabled cluster has a large count of nodes: 284 hosts. All nodes are using IPSEC communication successfully. One node however is not communicating with peers properly and ovnkube db rebuild is not sufficient to mitigate the behavior. Node remains out of sync with peers.

Observed in the sosreport the following libreswan log:

/home/remote/wrussell/04174538/0020-sosreport-lsvrocpp10139-2025-06-11-ggdiysr.tar.xz/sosreport-lsvrocpp10139-2025-06-11-ggdiysr/sos_commands/libreswan

certutil: function failed: SEC_ERROR_BAD_DATABASE: security library: bad database. 
certutil_-L_-d_sql_.etc.ipsec.d (END)

unhealthy node is taking upwards of 20s to reconcile:

2025-06-17T12:14:31.900545175Z 2025-06-17T12:14:31Z |1900167| ovs-monitor-ipsec | INFO | Refreshing LibreSwan configuration      
## <--- START OF RECONCILE 12:14:31.9
...<omitted>...
2025-06-17T12:14:49.643748671Z 2025-06-17T12:14:49Z |1900259| ovs-monitor-ipsec | INFO | Refreshing is done.                      
#  <--end of reconcile: 12:14:49.6
##averaging approx 20s!!##

Compare to healthy node in same cluster:
2025-06-17T14:02:46.244931968Z 2025-06-17T14:02:46Z |529963| ovs-monitor-ipsec | INFO | Active connections for port ovn-4d26f9-0 ['ovn-4d26f9-0-out-1'] do not match desired ['ovn-4d26f9-0-in-1', 'ovn-4d26f9-0-out-1'], need to reconcile 
2025-06-17T14:02:46.245671535Z 2025-06-17T14:02:46Z |529964| ovs-monitor-ipsec | INFO | Refreshing LibreSwan configuration 
...
2025-06-17T14:02:48.369542123Z 2025-06-17T14:02:48Z |529976| ovs-monitor-ipsec | INFO | Refreshing is done.  
##averaging around .02s##

(see extended log below)

Version-Release number of selected component (if applicable):

4.16.37

How reproducible:

• One node (ongoing).

Steps to Reproduce:

• Working on internal replicator - currently unable to replicate, appears localized to single node on target cluster which makes me suspect an issue with that hardware ( potentially disk i/o)

Actual results:

• Cluster IPSEC is impaired when traversing to or from this node.

Expected results:

• All hosts should reconcile together and not lose sync

Additional info:

• See sosreports and gather data in first private comment in the Jira below.

  //extended libreswan reconciliation log on working host:
  
  2025-06-17T14:02:13.724760428Z 2025-06-17T14:02:13Z |529938| ovs-monitor-ipsec | INFO | Refreshing LibreSwan configuration 
  2025-06-17T14:02:13.906092389Z 2025-06-17T14:02:13Z |529942| ovs-monitor-ipsec | INFO | ovn-4d26f9-0-in-1 is defunct, removing 
  2025-06-17T14:02:13.920730311Z 2025-06-17T14:02:13Z |529944| ovs-monitor-ipsec | INFO | ovn-4d26f9-0-out-1 is half-loaded, removing 
  2025-06-17T14:02:13.935866535Z 2025-06-17T14:02:13Z |529946| ovs-monitor-ipsec | INFO | Starting ipsec connection ovn-4d26f9-0-out-1 
  2025-06-17T14:02:14.917367478Z 2025-06-17T14:02:14Z |529948| ovs-monitor-ipsec | INFO | Adding ipsec connection ovn-4d26f9-0-in-1 
  2025-06-17T14:02:15.854668357Z 2025-06-17T14:02:15Z |529950| ovs-monitor-ipsec | INFO | Refreshing is done. 
  2025-06-17T14:02:30.946236016Z 2025-06-17T14:02:30Z |529953| ovs-monitor-ipsec | INFO | Active connections for port ovn-4d26f9-0 ['ovn-4d26f9-0-out-1'] do not match desired ['ovn-4d26f9-0-in-1', 'ovn-4d26f9-0-out-1'], need to reconcile 
  2025-06-17T14:02:30.946851840Z 2025-06-17T14:02:30Z |529954| ovs-monitor-ipsec | INFO | Refreshing LibreSwan configuration 
  2025-06-17T14:02:31.132163230Z 2025-06-17T14:02:31Z |529958| ovs-monitor-ipsec | INFO | Bringing up ipsec connection ovn-4d26f9-0-in-1 
  2025-06-17T14:02:31.152928268Z 2025-06-17T14:02:31Z |529960| ovs-monitor-ipsec | INFO | Refreshing is done. 
  2025-06-17T14:02:46.244931968Z 2025-06-17T14:02:46Z |529963| ovs-monitor-ipsec | INFO | Active connections for port ovn-4d26f9-0 ['ovn-4d26f9-0-out-1'] do not match de sired ['ovn-4d26f9-0-in-1', 'ovn-4d26f9-0-out-1'], need to reconcile 
  2025-06-17T14:02:46.245671535Z 2025-06-17T14:02:46Z |529964| ovs-monitor-ipsec | INFO | Refreshing LibreSwan configuration 
  2025-06-17T14:02:46.441040712Z 2025-06-17T14:02:46Z |529968| ovs-monitor-ipsec | INFO | ovn-4d26f9-0-in-1 is defunct, removing 
  2025-06-17T14:02:46.462256542Z 2025-06-17T14:02:46Z |529970| ovs-monitor-ipsec | INFO | ovn-4d26f9-0-out-1 is half-loaded, removing 
  2025-06-17T14:02:46.482285425Z 2025-06-17T14:02:46Z |529972| ovs-monitor-ipsec | INFO | Starting ipsec connection ovn-4d26f9-0-out-1 
  2025-06-17T14:02:47.419481019Z 2025-06-17T14:02:47Z |529974| ovs-monitor-ipsec | INFO | Adding ipsec connection ovn-4d26f9-0-in-1 
  2025-06-17T14:02:48.369542123Z 2025-06-17T14:02:48Z |529976| ovs-monitor-ipsec | INFO | Refreshing is done. 
  2025-06-17T14:03:03.468231000Z 2025-06-17T14:03:03Z |529979| ovs-monitor-ipsec | INFO | Active connections for port ovn-4d26f9-0 ['ovn-4d26f9-0-out-1'] do not match de sired ['ovn-4d26f9-0-in-1', 'ovn-4d26f9-0-out-1'], need to reconcile 
  2025-06-17T14:03:03.469029314Z 2025-06-17T14:03:03Z |529980| ovs-monitor-ipsec | INFO | Refreshing LibreSwan configuration 
  2025-06-17T14:03:03.671544745Z 2025-06-17T14:03:03Z |529984| ovs-monitor-ipsec | INFO | Bringing up ipsec connection ovn-4d26f9-0-in-1 
  2025-06-17T14:03:03.688589609Z 2025-06-17T14:03:03Z |529986| ovs-monitor-ipsec | INFO | Refreshing is done. 
  2025-06-17T14:03:18.780906947Z 2025-06-17T14:03:18Z |529989| ovs-monitor-ipsec | INFO | Active connections for port ovn-4d26f9-0 ['ovn-4d26f9-0-out-1'] do not match desired ['ovn-4d26f9-0-in-1', 'ovn-4d26f9-0-out-1'], need to reconcile  

  //extended libreswan reconciliation log on problem host:
  2025-06-17T12:14:31.900545175Z 2025-06-17T12:14:31Z |1900167| ovs-monitor-ipsec | INFO | Refreshing LibreSwan configuration ##<--- START OF RECONCILE 12:14:31.9 
  2025-06-17T12:14:32.067605854Z 2025-06-17T12:14:32Z |1900171| ovs-monitor-ipsec | INFO | ovn-994f62-0-in-1 is defunct, removing 
  2025-06-17T12:14:32.081952230Z 2025-06-17T12:14:32Z |1900173| ovs-monitor-ipsec | INFO | ovn-994f62-0-out-1 is half-loaded, removing 
  2025-06-17T12:14:32.096570828Z 2025-06-17T12:14:32Z |1900175| ovs-monitor-ipsec | INFO | Starting ipsec connection ovn-994f62-0-out-1 
  2025-06-17T12:14:33.057257761Z 2025-06-17T12:14:33Z |1900177| ovs-monitor-ipsec | INFO | Adding ipsec connection ovn-994f62-0-in-1 
  2025-06-17T12:14:33.978708867Z 2025-06-17T12:14:33Z |1900179| ovs-monitor-ipsec | INFO | ovn-218ee8-0-in-1 is defunct, removing 
  2025-06-17T12:14:33.992078310Z 2025-06-17T12:14:33Z |1900181| ovs-monitor-ipsec | INFO | ovn-218ee8-0-out-1 is half-loaded, removing 
  2025-06-17T12:14:34.005664620Z 2025-06-17T12:14:34Z |1900183| ovs-monitor-ipsec | INFO | Adding ipsec connection ovn-218ee8-0-in-1 
  2025-06-17T12:14:34.898508082Z 2025-06-17T12:14:34Z |1900185| ovs-monitor-ipsec | INFO | Starting ipsec connection ovn-218ee8-0-out-1 
  2025-06-17T12:14:35.842235834Z 2025-06-17T12:14:35Z |1900187| ovs-monitor-ipsec | INFO | Bringing up ipsec connection ovn-13abb7-0-in-1 
  2025-06-17T12:14:35.855353169Z 2025-06-17T12:14:35Z |1900189| ovs-monitor-ipsec | INFO | ovn-19c5a3-0-in-1 is defunct, removing 
  2025-06-17T12:14:35.868937431Z 2025-06-17T12:14:35Z |1900191| ovs-monitor-ipsec | INFO | ovn-19c5a3-0-out-1 is half-loaded, removing 
  2025-06-17T12:14:35.882929195Z 2025-06-17T12:14:35Z |1900193| ovs-monitor-ipsec | INFO | Starting ipsec connection ovn-19c5a3-0-out-1 
  2025-06-17T12:14:36.781081734Z 2025-06-17T12:14:36Z |1900195| ovs-monitor-ipsec | INFO | Adding ipsec connection ovn-19c5a3-0-in-1 
  2025-06-17T12:14:37.715040354Z 2025-06-17T12:14:37Z |1900197| ovs-monitor-ipsec | INFO | ovn-b8bbe1-0-in-1 is defunct, removing 
  2025-06-17T12:14:37.729151289Z 2025-06-17T12:14:37Z |1900199| ovs-monitor-ipsec | INFO | ovn-b8bbe1-0-out-1 is half-loaded, removing 
  2025-06-17T12:14:37.743474389Z 2025-06-17T12:14:37Z |1900201| ovs-monitor-ipsec | INFO | Starting ipsec connection ovn-b8bbe1-0-out-1 
  2025-06-17T12:14:38.711907439Z 2025-06-17T12:14:38Z |1900203| ovs-monitor-ipsec | INFO | Adding ipsec connection ovn-b8bbe1-0-in-1 
  2025-06-17T12:14:39.662740476Z 2025-06-17T12:14:39Z |1900205| ovs-monitor-ipsec | INFO | ovn-7034cf-0-in-1 is defunct, removing 
  2025-06-17T12:14:39.677366272Z 2025-06-17T12:14:39Z |1900207| ovs-monitor-ipsec | INFO | ovn-7034cf-0-out-1 is half-loaded, removing 
  2025-06-17T12:14:39.691940204Z 2025-06-17T12:14:39Z |1900209| ovs-monitor-ipsec | INFO | Starting ipsec connection ovn-7034cf-0-out-1 
  2025-06-17T12:14:40.683348426Z 2025-06-17T12:14:40Z |1900211| ovs-monitor-ipsec | INFO | Adding ipsec connection ovn-7034cf-0-in-1 
  2025-06-17T12:14:41.618816303Z 2025-06-17T12:14:41Z |1900213| ovs-monitor-ipsec | INFO | Bringing up ipsec connection ovn-5b900f-0-in-1 
  2025-06-17T12:14:41.632946270Z 2025-06-17T12:14:41Z |1900215| ovs-monitor-ipsec | INFO | Bringing up ipsec connection ovn-3fe292-0-in-1 
  2025-06-17T12:14:41.647427928Z 2025-06-17T12:14:41Z |1900217| ovs-monitor-ipsec | INFO | Bringing up ipsec connection ovn-ce9dc6-0-in-1 
  2025-06-17T12:14:41.661262609Z 2025-06-17T12:14:41Z |1900219| ovs-monitor-ipsec | INFO | ovn-b5d927-0-in-1 is defunct, removing 
  2025-06-17T12:14:41.675775525Z 2025-06-17T12:14:41Z |1900221| ovs-monitor-ipsec | INFO | ovn-b5d927-0-out-1 is half-loaded, removing 
  2025-06-17T12:14:41.689843908Z 2025-06-17T12:14:41Z |1900223| ovs-monitor-ipsec | INFO | Starting ipsec connection ovn-b5d927-0-out-1 
  2025-06-17T12:14:42.720899746Z 2025-06-17T12:14:42Z |1900225| ovs-monitor-ipsec | INFO | Adding ipsec connection ovn-b5d927-0-in-1 
  2025-06-17T12:14:43.713810890Z 2025-06-17T12:14:43Z |1900227| ovs-monitor-ipsec | INFO | Bringing up ipsec connection ovn-ec146a-0-in-1 
  2025-06-17T12:14:43.727684797Z 2025-06-17T12:14:43Z |1900229| ovs-monitor-ipsec | INFO | ovn-18c555-0-in-1 is defunct, removing 
  2025-06-17T12:14:43.741835900Z 2025-06-17T12:14:43Z |1900231| ovs-monitor-ipsec | INFO | ovn-18c555-0-out-1 is half-loaded, removing 
  2025-06-17T12:14:43.755839691Z 2025-06-17T12:14:43Z |1900233| ovs-monitor-ipsec | INFO | Starting ipsec connection ovn-18c555-0-out-1 
  2025-06-17T12:14:44.716706982Z 2025-06-17T12:14:44Z |1900235| ovs-monitor-ipsec | INFO | Adding ipsec connection ovn-18c555-0-in-1 
  2025-06-17T12:14:45.662377141Z 2025-06-17T12:14:45Z |1900237| ovs-monitor-ipsec | INFO | ovn-6b5fb4-0-in-1 is defunct, removing 
  2025-06-17T12:14:45.676102245Z 2025-06-17T12:14:45Z |1900239| ovs-monitor-ipsec | INFO | ovn-6b5fb4-0-out-1 is half-loaded, removing 
  2025-06-17T12:14:45.690477580Z 2025-06-17T12:14:45Z |1900241| ovs-monitor-ipsec | INFO | Adding ipsec connection ovn-6b5fb4-0-in-1 
  2025-06-17T12:14:46.706576076Z 2025-06-17T12:14:46Z |1900243| ovs-monitor-ipsec | INFO | Starting ipsec connection ovn-6b5fb4-0-out-1 
  2025-06-17T12:14:47.673265309Z 2025-06-17T12:14:47Z |1900245| ovs-monitor-ipsec | INFO | Bringing up ipsec connection ovn-875c25-0-in-1 
  2025-06-17T12:14:47.686850084Z 2025-06-17T12:14:47Z |1900247| ovs-monitor-ipsec | INFO | Bringing up ipsec connection ovn-429d97-0-in-1 
  2025-06-17T12:14:47.700994145Z 2025-06-17T12:14:47Z |1900249| ovs-monitor-ipsec | INFO | Bringing up ipsec connection ovn-239f85-0-in-1 
  2025-06-17T12:14:47.716560591Z 2025-06-17T12:14:47Z |1900251| ovs-monitor-ipsec | INFO | ovn-9c8c00-0-in-1 is defunct, removing 
  2025-06-17T12:14:47.733415231Z 2025-06-17T12:14:47Z |1900253| ovs-monitor-ipsec | INFO | ovn-9c8c00-0-out-1 is half-loaded, removing 
  2025-06-17T12:14:47.749655428Z 2025-06-17T12:14:47Z |1900255| ovs-monitor-ipsec | INFO | Adding ipsec connection ovn-9c8c00-0-in-1 
  2025-06-17T12:14:48.706952343Z 2025-06-17T12:14:48Z |1900257| ovs-monitor-ipsec | INFO | Starting ipsec connection ovn-9c8c00-0-out-1 
  2025-06-17T12:14:49.643748671Z 2025-06-17T12:14:49Z |1900259| ovs-monitor-ipsec | INFO | Refreshing is done.                      #<--end of reconcile: 12:14:49.6 
  2025-06-17T12:15:04.722120736Z 2025-06-17T12:15:04Z |1900262| ovs-monitor-ipsec | INFO | Active connections for port ovn-994f62-0 ['ovn-994f62-0-out-1'] do not match desired ['ovn-994f62-0-in-1', 'ovn-994f62-0-out-1'], need to reconcile  

  Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.