Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-11046

TuningCNI cnf-test failure: sysctl allowlist update

    • No
    • CNF Network Sprint 234, CNF Network Sprint 235, CNF Network Sprint 236, CNF Network Sprint 237
    • 4
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Currently, users cannot modify the `interface-specific` safe sysctl list by updating the `cni-sysctl-allowlist` config map in the `openshift-multus` namespace. As a workaround, you can modify, either manually or with a Daemon Set, the file `/etc/cni/tuning/allowlist.conf` on the node or nodes. (link:https://issues.redhat.com/browse/OCPBUGS-11046[*OCPBUGS-11046*])
      Show
      * Currently, users cannot modify the `interface-specific` safe sysctl list by updating the `cni-sysctl-allowlist` config map in the `openshift-multus` namespace. As a workaround, you can modify, either manually or with a Daemon Set, the file `/etc/cni/tuning/allowlist.conf` on the node or nodes. (link: https://issues.redhat.com/browse/OCPBUGS-11046 [* OCPBUGS-11046 *])
    • Known Issue
    • Proposed

      Description of problem:

      The following test is permafeailing in Prow CI:
      [tuningcni] sysctl allowlist update [It] should start a pod with custom sysctl only after adding sysctl to allowlist
      
      https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-kni-cnf-features-deploy-master-e2e-gcp-ovn-periodic/1640987392103944192
      
      
      [tuningcni]
      9915/go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/e2esuite/security/tuning.go:26
      9916  sysctl allowlist update
      9917  /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/e2esuite/security/tuning.go:141
      9918    should start a pod with custom sysctl only after adding sysctl to allowlist
      9919    /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/e2esuite/security/tuning.go:156
      9920  > Enter [BeforeEach] [tuningcni] - /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/pkg/execute/ginkgo.go:9 @ 03/29/23 10:08:49.855
      9921  < Exit [BeforeEach] [tuningcni] - /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/pkg/execute/ginkgo.go:9 @ 03/29/23 10:08:49.855 (0s)
      9922  > Enter [BeforeEach] sysctl allowlist update - /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/e2esuite/security/tuning.go:144 @ 03/29/23 10:08:49.855
      9923  < Exit [BeforeEach] sysctl allowlist update - /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/e2esuite/security/tuning.go:144 @ 03/29/23 10:08:49.896 (41ms)
      9924  > Enter [It] should start a pod with custom sysctl only after adding sysctl to allowlist - /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/e2esuite/security/tuning.go:156 @ 03/29/23 10:08:49.896
      9925  [FAILED] Unexpected error:
      9926      <*errors.errorString | 0xc00044eec0>: {
      9927          s: "timed out waiting for the condition",
      9928      }
      9929      timed out waiting for the condition
      9930  occurred9931  In [It] at: /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/e2esuite/security/tuning.go:186 @ 03/29/23 10:09:53.377
      
      

      Version-Release number of selected component (if applicable):

      master (4.14)

      How reproducible:

       

      Steps to Reproduce:

      1.
      2.
      3.
      

      Actual results:

      Test fails

      Expected results:

      Test passes

      Additional info:

      PR https://github.com/openshift-kni/cnf-features-deploy/pull/1445 adds some useful information to the reported archive.

            [OCPBUGS-11046] TuningCNI cnf-test failure: sysctl allowlist update

            Errata Tool added a comment -

            Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

            For information on the advisory (Important: OpenShift Container Platform 4.14.0 bug fix and security update), and where to find the updated files, follow the link below.

            If the solution does not work for you, open a new bug report.
            https://access.redhat.com/errata/RHSA-2023:5006

            Errata Tool added a comment - Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Important: OpenShift Container Platform 4.14.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:5006

            Hi all,

            I ran the tests from the new image cnf-tests:v4.13.1-6.

            The test was run and passed.

            QA bug validated

            running /usr/bin/configsuite -ginkgo.focus=sysctl allowlist update should start a pod with custom sysctl only after adding sysctl to allowlist
            I0531 11:56:27.151700      27 request.go:690] Waited for 1.046405959s due to client-side throttling, not priority and fairness, request: GET:https://api.hlxcl7.lab.eng.tlv2.redhat.com:6443/apis/quota.openshift.io/v1?timeout=32s
            Running Suite: CNF Features e2e setup - /
            =========================================
            Random Seed: 1685534185

            Will run 0 of 2 specs
            SS

            Ran 0 of 2 Specs in 0.000 seconds
            SUCCESS! – 0 Passed | 0 Failed | 0 Pending | 2 Skipped
            PASS
            running /usr/bin/cnftests -ginkgo.focus=sysctl allowlist update should start a pod with custom sysctl only after adding sysctl to allowlist
            I0531 11:56:30.305859      47 request.go:690] Waited for 1.044478963s due to client-side throttling, not priority and fairness, request: GET:https://api.hlxcl7.lab.eng.tlv2.redhat.com:6443/apis/quota.openshift.io/v1?timeout=32s
            time="2023-05-31T11:56:32Z" level=info msg="MAX_OFFSET_IN_NS not set, assuming 100 ns"
            time="2023-05-31T11:56:32Z" level=info msg="MIN_OFFSET_IN_NS not set, assuming -100 ns"
            Running Suite: CNF Features e2e integration tests - /
            =====================================================
            Random Seed: 1685534188

            Will run 1 of 226 specs
            SSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS

            Ran 1 of 226 Specs in 65.131 seconds
            SUCCESS! – 1 Passed | 0 Failed | 0 Pending | 225 Skipped

            Gregory Kopels added a comment - Hi all, I ran the tests from the new image cnf-tests:v4.13.1-6. The test was run and passed. QA bug validated running /usr/bin/configsuite -ginkgo.focus=sysctl allowlist update should start a pod with custom sysctl only after adding sysctl to allowlist I0531 11:56:27.151700      27 request.go:690] Waited for 1.046405959s due to client-side throttling, not priority and fairness, request: GET: https://api.hlxcl7.lab.eng.tlv2.redhat.com:6443/apis/quota.openshift.io/v1?timeout=32s Running Suite: CNF Features e2e setup - / ========================================= Random Seed: 1685534185 Will run 0 of 2 specs SS Ran 0 of 2 Specs in 0.000 seconds SUCCESS! – 0 Passed | 0 Failed | 0 Pending | 2 Skipped PASS running /usr/bin/cnftests -ginkgo.focus=sysctl allowlist update should start a pod with custom sysctl only after adding sysctl to allowlist I0531 11:56:30.305859      47 request.go:690] Waited for 1.044478963s due to client-side throttling, not priority and fairness, request: GET: https://api.hlxcl7.lab.eng.tlv2.redhat.com:6443/apis/quota.openshift.io/v1?timeout=32s time="2023-05-31T11:56:32Z" level=info msg="MAX_OFFSET_IN_NS not set, assuming 100 ns" time="2023-05-31T11:56:32Z" level=info msg="MIN_OFFSET_IN_NS not set, assuming -100 ns" Running Suite: CNF Features e2e integration tests - / ===================================================== Random Seed: 1685534188 Will run 1 of 226 specs SSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS Ran 1 of 226 Specs in 65.131 seconds SUCCESS! – 1 Passed | 0 Failed | 0 Pending | 225 Skipped

            apanatto@redhat.com, I'll investigate this

            Marcelo Guerrero Viveros added a comment - apanatto@redhat.com , I'll investigate this

            In [1], there are several red herrings errors:

              [FAIL] validation [sctp] [It] should have the sctp enable machine config as part of the CNF machine config pool2500 

            They are about a wait loop for features to be installed on the cluster.

            The real problem with that job is at the end:

            Summarizing 6 Failures:
            10650  [FAIL] [rfe_id:27368][performance] RPS configuration [It] [test_id: 59572] Check RPS Mask is applied to atleast one single rx queue on all veth interface
            10651  /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/1_performance/performance.go:323
            10652  [FAIL] [performance] Checking IRQBalance settings Verify irqbalance configuration handling [It] Should not overwrite the banned CPU set on tuned restart
            10653  /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/1_performance/irqbalance.go:226
            10654  [FAIL] [xt_u32] Validate the module is enabled and works [It] Should create an iptables rule inside a pod that has the module enabled
            10655  /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/e2esuite/xt_u32/xt_u32.go:155
            10656  [FAIL] [tuningcni] tuningcni over bond [It] pods with sysctls over bond should be able to ping each other
            10657  /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/e2esuite/security/tuning.go:137
            10658  [FAIL] [tuningcni] sysctl allowlist update [It] should start a pod with custom sysctl only after adding sysctl to allowlist
            10659  /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/e2esuite/security/tuning.go:186
            10660  [FAIL] [rfe_id:27363][performance] CPU Management when pod runs with the CPU load balancing runtime class [It] [test_id:32646] should disable CPU load balancing for CPU's used by the pod
            10661  /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/1_performance/cpu_management.go:310 

             

            I filed a PR [2] to better organize the JUnit result in the Prow Job page, placing them in the JUnit section.

            Issues [3] and [4] chase performance test failures.

            [1] https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-kni-cnf-features-deploy-master-e2e-gcp-ovn-periodic/1640987392103944192
            [2] https://github.com/openshift-kni/cnf-features-deploy/pull/1444 
            [3] https://issues.redhat.com/browse/OCPBUGS-10494
            [4] https://issues.redhat.com/browse/OCPNODE-1538

            Andrea Panattoni added a comment - In [1] , there are several red herrings errors: [FAIL] validation [sctp] [It] should have the sctp enable machine config as part of the CNF machine config pool2500 They are about a wait loop for features to be installed on the cluster. The real problem with that job is at the end: Summarizing 6 Failures: 10650 [FAIL] [rfe_id:27368][performance] RPS configuration [It] [test_id: 59572] Check RPS Mask is applied to atleast one single rx queue on all veth interface 10651 /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/1_performance/performance.go:323 10652 [FAIL] [performance] Checking IRQBalance settings Verify irqbalance configuration handling [It] Should not overwrite the banned CPU set on tuned restart 10653 /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/1_performance/irqbalance.go:226 10654 [FAIL] [xt_u32] Validate the module is enabled and works [It] Should create an iptables rule inside a pod that has the module enabled 10655 /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/e2esuite/xt_u32/xt_u32.go:155 10656 [FAIL] [tuningcni] tuningcni over bond [It] pods with sysctls over bond should be able to ping each other 10657 /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/e2esuite/security/tuning.go:137 10658 [FAIL] [tuningcni] sysctl allowlist update [It] should start a pod with custom sysctl only after adding sysctl to allowlist 10659 /go/src/github.com/openshift-kni/cnf-features-deploy/cnf-tests/testsuites/e2esuite/security/tuning.go:186 10660 [FAIL] [rfe_id:27363][performance] CPU Management when pod runs with the CPU load balancing runtime class [It] [test_id:32646] should disable CPU load balancing for CPU's used by the pod 10661 /go/src/github.com/openshift-kni/cnf-features-deploy/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/1_performance/cpu_management.go:310   I filed a PR [2] to better organize the JUnit result in the Prow Job page, placing them in the JUnit section. Issues [3] and [4] chase performance test failures. [1] https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-kni-cnf-features-deploy-master-e2e-gcp-ovn-periodic/1640987392103944192 [2] https://github.com/openshift-kni/cnf-features-deploy/pull/1444   [3] https://issues.redhat.com/browse/OCPBUGS-10494 [4] https://issues.redhat.com/browse/OCPNODE-1538

            the prow link shows a SCTP failure. apanatto@redhat.com are you sure it's the right link?

            Yuval Kashtan added a comment - the prow link shows a SCTP failure. apanatto@redhat.com are you sure it's the right link?

              rh-ee-marguerr Marcelo Guerrero Viveros
              apanatto@redhat.com Andrea Panattoni
              Gregory Kopels Gregory Kopels
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated:
                Resolved: