Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-32018

NodeHasIntegrityFailure alert due to multus certificate rotation

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • 4.14.z
    • None
    • No
    • Proposed
    • False
    • Hide

      None

      Show
      None

      On an OpenShift 4.14.11, we were receiving NodeHasIntegrityFailure alerts. We jumped into the aide pod and performed the aide integrity check manually:

      $ aide --config /config/aide.conf
      Start timestamp: 2024-04-04 17:12:00 +0000 (AIDE 0.16)
      AIDE found differences between database and filesystem!!
       
      Summary:
        Total number of entries:      32432
        Added entries:                1
        Removed entries:              0
        Changed entries:              0
       
      ---------------------------------------------------
      Added entries:
      ---------------------------------------------------
       
      f++++++++++++++++: /hostroot/etc/cni/multus/certs/multus-client-2024-04-04-02-48-51.pem
       
      ---------------------------------------------------
      The attributes of the (uncompressed) database(s):
      ---------------------------------------------------
       
      /hostroot/etc/kubernetes/aide.db.gz
        MD5      : QsgWwipQaL1BErcS1u97oA==
        SHA1     : NSbj7T0JJ+CPfQ3zhHDzMnh4ZJw=
        RMD160   : cp/cIGbiCGlc8C52BuqMXUVItm4=
        TIGER    : dFYfqeO6SzeI08fTRhM6RVBB+NKlmHST
        SHA256   : k+ZA8FcsJxa+6sZuJEc2v7J4pyRVmqSu
                   972PvldfcSE=
        SHA512   : 7LdGG2SnQbn5MxZrKYtp2aD15gQCXKnZ
                   yfPNirUYYunfRQmQ758BoUEJA69bbgcL
                   bMan1Ya++Vx5SpSvtTZTlw==
       
      End timestamp: 2024-04-04 17:12:44 +0000 (run time: 0m 44s) 

      The AIDE integrity check failed due to multus certificate files that are being created on the cluster nodes:

      $ ls -1 /etc/cni/multus/certs
      ...
      multus-client-2024-04-05-12-59-47.pem
      multus-client-2024-04-06-14-16-17.pem
      multus-client-2024-04-07-13-32-45.pem
      multus-client-2024-04-08-13-02-33.pem
      multus-client-2024-04-09-13-09-53.pem
      multus-client-current.pem

      To fix the issue, we created a custom aide.conf. This custom aide.conf included the original configuration and we added a line that instructed AIDE to ignore the multus certificate files:

       ...
      # Ignore multus certificate rotation
      !/hostroot/etc/cni/multus/certs/multus-client-.*.pem
      ...

      The above configuration fixed the issue for us. Please, consider including the above rule in the default aide.conf.

            wenshen@redhat.com Vincent Shen
            anosek@redhat.com Ales Nosek
            Xiaojie Yuan Xiaojie Yuan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: