The cnf-tests image is used by customers and internally to run latency tests - see https://docs.openshift.com/container-platform/4.16/scalability_and_performance/low_latency_tuning/cnf-performing-platform-verification-latency-tests.html

When the tests are run by the user, pods are created on the SUT that pull the cnf-tests image to run the tests. These pods are pulling the wrong image version. For example, running the tests with cnf-tests:4.16 will result in the cnf-tests:4.9 image being used on the SUT. The cnf-tests:4.9 image has bugs in it that can cause test failures (e.g. OCPBUGS-2618).

Here is an example command that would show the issue:

podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig.yaml -e LATENCY_TEST_CPUS=10 -e LATENCY_TEST_RUNTIME=300 -e MAXIMUM_LATENCY=20 quay.io/openshift-kni/cnf-tests:4.16 /usr/bin/test-run.sh --ginkgo.focus="cyclictest" --ginkgo.v --ginkgo.timeout="24h"

Checking the cyclictest-xxxxx pod created on the SUT, I see:

image: quay.io/openshift-kni/cnf-tests:4.9

There is a workaround - the IMAGE_REGISTRY and CNF_TESTS_IMAGE can also be specified in the command, but those are optional and most users are not using them.

This is customer impacting as many of them run latency tests using this image, so I think this needs to be fixed all the way back to the last supported release (probably OCP 4.14?).