Uploaded image for project: 'Fast Datapath Product'
  1. Fast Datapath Product
  2. FDP-1473 Re-starting ovs-monitor-ipsec with --no-restart-ike-daemon causes cert duplication in NSS database
  3. FDP-1505

[RHEL-9 OVS-3.4] Re-starting ovs-monitor-ipsec with --no-restart-ike-daemon causes cert duplication in NSS database

    • Icon: Sub-task Sub-task
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • None
    • openvswitch3.4
    • None
    • 0
    • False
    • Hide

      None

      Show
      None
    • False
    • openvswitch3.4-3.4.3-82.el9fdp
    • rhel-9
    • rhel-net-ovs-dpdk
    • ssg_networking

       Problem Description: Clearly explain the issue.

      When ovs-monitor-ipsec daemon starts normally, it will remove all OVS-related certs and keys from the NSS database. This way, when it discovers ports and creates connections for them, it will import current certs and keys and these will be the only ones in the NSS database. However, if the daemon is stopped, then the certs changed on disk and then the daemon is restarted with --no-restart-ike-daemon, the daemon will not clear the database and then will attempt to add new certs and keys for existing ports. Since these certs were changed on disk, they will be added to the database. This way we end up with multiple certs having the same nickname. This causes connections to randomly use the old outdated certificate and fail to be established.
       

       Impact Assessment: Describe the severity and impact (e.g., network down,availability of a workaround, etc.).

      Causes connectivity issues in OpenShift when certificate files are updated.
       

       Software Versions: Specify the exact versions in use (e.g.,openvswitch3.1-3.1.0-147.el8fdp).

      openvswitch3.5-3.5.0-19.el9fdp
       

        Issue Type: Indicate whether this is a new issue or a regression (if a regression, state the last known working version).

      New issue.
       

       Reproducibility: Confirm if the issue can be reproduced consistently. If not, describe how often it occurs.

      100%
       

       Reproduction Steps: Provide detailed steps or scripts to replicate the issue.

      1. Configure IPsec.
      2. Kill ovs-monitor-ipsec.
      3. Update certificates.
      4. Re-start ovs-monitor-ipsec with --no-restart-ike-daemon.
       

       Expected Behavior: Describe what should happen under normal circumstances.

      Old certs and keys should not be in the NSS database.
       

       Observed Behavior: Explain what actually happens.

      Duplicate certs in the NSS database, connections fail to be established:

      sh-5.1# certutil -L -d sql:/var/lib/ipsec/nss 
      
      Certificate Nickname                                         Trust Attributes
                                                                   SSL,S/MIME,JAR/XPI
      
      ovs_certkey_c04c352b-d738-4e38-a8e5-7255f494e543             u,u,u
      ovs_cert_cacert                                              CT,, 
      ovs_certkey_c04c352b-d738-4e38-a8e5-7255f494e543             u,u,u
      ovs_certkey_c04c352b-d738-4e38-a8e5-7255f494e543             u,u,u
      ovs_certkey_c04c352b-d738-4e38-a8e5-7255f494e543             u,u,u
      ovs_certkey_c04c352b-d738-4e38-a8e5-7255f494e543             u,u,u
      ovs_certkey_c04c352b-d738-4e38-a8e5-7255f494e543             u,u,u
      ovs_certkey_c04c352b-d738-4e38-a8e5-7255f494e543             u,u,u
      
      Jun 05 08:37:58 worker-0 pluto[1636723]: "ovn-c04c35-0-out-1" #459: processing decrypted IKE_AUTH request from UDP/500 containing SK{IDi,CERT,CERTREQ,IDr,AUTH,SA,TSi,TSr,N(USE_TRANSPORT_MODE)}
      Jun 05 08:37:58 worker-0 pluto[1636723]: "ovn-c04c35-0-out-1" #459: NSS: ERROR: IPsec certificate CN=c04c352b-d738-4e38-a8e5-7255f494e543,OU=kind,O=ovnkubernetes,C=US invalid: SEC_ERROR_UNKNOWN_ISSUER: Peer's Certificate issuer is not recognized.
      Jun 05 08:37:58 worker-0 pluto[1636723]: "ovn-c04c35-0-out-1" #459: NSS: end certificate invalid
      

      Slack thread: https://redhat-internal.slack.com/archives/C08DNAFC85T/p1749114943661179

              imaximet@redhat.com Ilya Maximets
              imaximet@redhat.com Ilya Maximets
              Jiying Qiu Jiying Qiu
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: