-
Bug
-
Resolution: Unresolved
-
Minor
-
CNV v4.21.0, CNV v4.21.z
-
None
-
Product / Portfolio Work
-
0.42
-
False
-
-
False
-
None
-
-
CNV I/U Operators Sprint 283
-
None
Description of problem:
The validation checkup pod shows a misleading "unauthorized" error when the virt-operator image doesn't exist in the mirror registry during disconnected environment deployments. The error message suggests an authentication problem, but the real issue is that the image is missing. This causes users to waste time troubleshooting authentication when that's not the problem.
$ oc logs -n ocp-virt-validation ocp-virt-validation-job-20260130-125920-g9nqz Cluster "in-cluster" set. User "sa-user" set. Context "sa-context" created. Switched to context "sa-context". Adding owner reference to PVC ocp-virt-validation-pvc-20260130-125920... Found owner Job: ocp-virt-validation-job-20260130-125920 (UID: 920f981f-2494-4423-a118-39d1564ef445) persistentvolumeclaim/ocp-virt-validation-pvc-20260130-125920 patched Successfully added owner reference to PVC ocp-virt-validation-pvc-20260130-125920 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 29.2M 100 29.2M 0 0 201M 0 --:--:-- --:--:-- --:--:-- 201M virtctl downloaded Replacing registry server with: cnv-qe-infra-01.cnvqe2.lab.eng.rdu2.redhat.com:8443 Using virt-operator image: cnv-qe-infra-01.cnvqe2.lab.eng.rdu2.redhat.com:8443/container-native-virtualization/virt-operator-rhel9@sha256:40c8fcd488c8bc43428cba11c8094fd7eb8cbf255314e806acdaf97b93ba8b1e error: image "cnv-qe-infra-01.cnvqe2.lab.eng.rdu2.redhat.com:8443/container-native-virtualization/virt-operator-rhel9@sha256:40c8fcd488c8bc43428cba11c8094fd7eb8cbf255314e806acdaf97b93ba8b1e" not found: manifest u nknown: manifest unknown error: unable to read image brew.cnv-qe-infra-01.cnvqe2.lab.eng.rdu2.redhat.com:8443/container-native-virtualization/virt-operator-rhel9@sha256:40c8fcd488c8bc43428cba11c8094fd7eb8cbf255314e806acdaf97b93ba8b1e: Ge t "http://brew.cnv-qe-infra-01.cnvqe2.lab.eng.rdu2.redhat.com:8443/v2/": dial tcp: lookup brew.cnv-qe-infra-01.cnvqe2.lab.eng.rdu2.redhat.com on 172.30.0.10:53: no such host Error: could not get kubevirt tag from virt-operator image.
Version-Release number of selected component (if applicable):
OpenShift Virtualization 4.21 (pre-GA) ocp-virt-validation-checkup
How reproducible:
Always - can be consistently reproduced when running validation checkup in disconnected environment with virt-operator image missing from mirror registry
Steps to Reproduce:
1. Set up a disconnected cluster with OpenShift Virtualization installed 2. Set up a local mirror registry with proper TLS trust and authentication configured 3. Mirror the validation checkup image to the registry, but do NOT mirror the virt-operator image 4. Run validation checkup with REGISTRY_SERVER parameter pointing to mirror registry 5. Observe the error in the pod logs
Actual results:
The validation checkup fails with this error:
Using virt-operator image: cnv-qe-infra-01.cnvqe2.lab.eng.rdu2.redhat.com:8443/container-native-virtualization/virt-operator-rhel9@sha256:40c8fcd488c8bc43428cba11c8094fd7eb8cbf255314e806acdaf97b93ba8b1e error: unable to read image cnv-qe-infra-01.cnvqe2.lab.eng.rdu2.redhat.com:8443/container-native-virtualization/virt-operator-rhel9@sha256:40c8fcd488c8bc43428cba11c8094fd7eb8cbf255314e806acdaf97b93ba8b1e: unauthorized: access to the requested resource is not authorized
This error message is misleading because it suggests an authentication/credentials problem when the real issue is that the image manifest doesn't exist in the registry.
Expected results:
The error should clearly indicate the image doesn't exist, either as:
{code:bash}
error: unable to read image [...]: manifest unknown
Or with a more descriptive message:
error: virt-operator image not found in registry. In disconnected environments, ensure the virt-operator image is mirrored to your local registry.
Additional info:
Root Cause Analysis:
The oc image info command in scripts/entrypoint.sh (lines 161-164) returns "unauthorized" when it can't find the image manifest, even though the real issue is the manifest doesn't exist rather than an
authentication failure.
KUBEVIRT_TAG=$(oc image info -a ${REGISTRY_CONFIG} ${INSECURE_FLAG} ${VIRT_OPERATOR_IMAGE} -o json --filter-by-os=linux/amd64 | jq -r '.config.config.Labels["upstream-version"]')
if [ -z "${KUBEVIRT_TAG}" ]
then
KUBEVIRT_TAG=$(oc image info -a ${REGISTRY_CONFIG} ${INSECURE_FLAG} brew.${VIRT_OPERATOR_IMAGE} -o json --filter-by-os=linux/amd64 | jq -r '.config.config.Labels["upstream-version"]')
fi
Impact:
This affects users setting up validation checkup in disconnected environments:
- Wastes significant troubleshooting time investigating authentication when that's not the problem
- Confuses users who have already properly configured authentication and TLS
- Not documented that virt-operator image needs to be mirrored
- README.md mentions mirroring test images but doesn't mention virt-operator requirement
Suggested Fix:
Add error handling to check if virt-operator image exists and provide clear guidance:
KUBEVIRT_TAG=$(oc image info -a ${REGISTRY_CONFIG} ${INSECURE_FLAG} ${VIRT_OPERATOR_IMAGE} -o json --filter-by-os=linux/amd64 2>&1 | jq -r '.config.config.Labels["upstream-version"]')
if [ -z "${KUBEVIRT_TAG}" ]
then
# Check if image exists before trying brew. prefix
if ! oc image info -a ${REGISTRY_CONFIG} ${INSECURE_FLAG} ${VIRT_OPERATOR_IMAGE} &>/dev/null; then
echo "Error: virt-operator image not found: ${VIRT_OPERATOR_IMAGE}"
if [ -n "${REGISTRY_SERVER}" ]; then
echo "In disconnected environments, ensure the virt-operator image is mirrored:"
echo " Original image: $(oc get deployment virt-operator -n openshift-cnv -o jsonpath='{.spec.template.spec.containers[0].image}')"
echo " Mirror command: oc image mirror --insecure=true <original-image> ${REGISTRY_SERVER}/container-native-virtualization/virt-operator-rhel9"
fi
exit 1
fi
KUBEVIRT_TAG=$(oc image info -a ${REGISTRY_CONFIG} ${INSECURE_FLAG} brew.${VIRT_OPERATOR_IMAGE} -o json --filter-by-os=linux/amd64 | jq -r '.config.config.Labels["upstream-version"]')
fi
Workaround:
Mirror the virt-operator image to the local registry:
VIRT_OPERATOR_IMAGE=$(oc get deployment virt-operator -n openshift-cnv -o jsonpath='{.spec.template.spec.containers[0].image}') # For IIB-based installations, use brew. prefix oc image mirror \ --registry-config=pull_secret.json \ --insecure=true \ brew.${VIRT_OPERATOR_IMAGE} \ ${REGISTRY_HOST}/container-native-virtualization/virt-operator-rhel9
- is triggered by
-
CNV-68306 OCP Virtualization self-certification package for Cloud providers (GA) - UI and disconnected T1
-
- In Progress
-