-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
4.18
-
None
-
Moderate
-
None
-
False
-
Description of problem:
There are recently 6 runs of periodic-ci-openshift-release-master-nightly-4.18-e2e-metal-ipi-upgrade-ovn-ipv6 that failed on the test suites "openshift-tests-upgrade" and "Cluster upgrade" when the cluster version operator had troubles to retrieve the payload from build05's registry. Those jobs caught the OTA team's attention because it impacts the readiness for cluster-version-operator in Sippy.
The failed tests are:
openshift-tests-upgrade.[sig-arch][Feature:ClusterUpgrade] Cluster should be upgradeable after finishing upgrade [Late][Suite:upgrade]
openshift-tests-upgrade.[sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]
Cluster upgrade.[sig-cluster-lifecycle] Cluster completes upgrade
Cluster upgrade.[sig-cluster-lifecycle] Cluster version operator acknowledges upgrade
The links to the Prow jobs:
The events collected in the job's artifacts indicate that two types of failures:
- "manifest unknown" for images in build05's registry (all 6 of them).
- "network is unreachable" feels like a networking issue on the cluster (3 of them).
find . -name events -type f | while read file; do rg 'openshift-cluster-version.*pod/version-.*Failed' $file | tail -n 1 ; done openshift-cluster-version 8m21s Warning Failed pod/version--xptdp-65qzx Failed to pull image "registry.build05.ci.openshift.org/ci-op-mirp2qrc/release@sha256:32520b17d1fedc28b346a523ebddee4859570310feb89b98026c688e6d8f1dfb": initializing source docker://registry.build05.ci.openshift.org/ci-op-mirp2qrc/release@sha256:32520b17d1fedc28b346a523ebddee4859570310feb89b98026c688e6d8f1dfb: (Mirrors also failed: [virthost.ostest.test.metalkube.org:5000/localimages/local-release-image@sha256:32520b17d1fedc28b346a523ebddee4859570310feb89b98026c688e6d8f1dfb: reading manifest sha256:32520b17d1fedc28b346a523ebddee4859570310feb89b98026c688e6d8f1dfb in virthost.ostest.test.metalkube.org:5000/localimages/local-release-image: manifest unknown]): registry.build05.ci.openshift.org/ci-op-mirp2qrc/release@sha256:32520b17d1fedc28b346a523ebddee4859570310feb89b98026c688e6d8f1dfb: pinging container registry registry.build05.ci.openshift.org: Get "https://registry.build05.ci.openshift.org/v2/": dial tcp 3.212.211.54:443: connect: network is unreachable openshift-cluster-version 39s Warning Failed pod/version--xsgp4-6mj47 Failed to pull image "registry.build05.ci.openshift.org/ci-op-2nlw8bbl/release@sha256:e87cdacdf5c575ff99d4cca7ec38758512a408ac1653fef8dd7b2c4b85e295f4": initializing source docker://registry.build05.ci.openshift.org/ci-op-2nlw8bbl/release@sha256:e87cdacdf5c575ff99d4cca7ec38758512a408ac1653fef8dd7b2c4b85e295f4: (Mirrors also failed: [virthost.ostest.test.metalkube.org:5000/localimages/local-release-image@sha256:e87cdacdf5c575ff99d4cca7ec38758512a408ac1653fef8dd7b2c4b85e295f4: reading manifest sha256:e87cdacdf5c575ff99d4cca7ec38758512a408ac1653fef8dd7b2c4b85e295f4 in virthost.ostest.test.metalkube.org:5000/localimages/local-release-image: manifest unknown]... openshift-cluster-version 12m Warning Failed pod/version--sbqg2-znml5 Failed to pull image "registry.build05.ci.openshift.org/ci-op-7cz9ls6d/release@sha256:9dbc3600441a49906c3abb6f3e27fe1d4c8d03769c69c15157a4a7a69d92cc06": initializing source docker://registry.build05.ci.openshift.org/ci-op-7cz9ls6d/release@sha256:9dbc3600441a49906c3abb6f3e27fe1d4c8d03769c69c15157a4a7a69d92cc06: (Mirrors also failed: [virthost.ostest.test.metalkube.org:5000/localimages/local-release-image@sha256:9dbc3600441a49906c3abb6f3e27fe1d4c8d03769c69c15157a4a7a69d92cc06: reading manifest sha256:9dbc3600441a49906c3abb6f3e27fe1d4c8d03769c69c15157a4a7a69d92cc06 in virthost.ostest.test.metalkube.org:5000/localimages/local-release-image: manifest unknown]... openshift-cluster-version 9m45s Warning Failed pod/version--qzflb-gdh7v Failed to pull image "registry.build05.ci.openshift.org/ci-op-wxyv2cs7/release@sha256:f457cec1b76f146536d0203d91d78777f7759022e6c665fe2c1e8b5ef705ba75": initializing source docker://registry.build05.ci.openshift.org/ci-op-wxyv2cs7/release@sha256:f457cec1b76f146536d0203d91d78777f7759022e6c665fe2c1e8b5ef705ba75: (Mirrors also failed: [virthost.ostest.test.metalkube.org:5000/localimages/local-release-image@sha256:f457cec1b76f146536d0203d91d78777f7759022e6c665fe2c1e8b5ef705ba75: reading manifest sha256:f457cec1b76f146536d0203d91d78777f7759022e6c665fe2c1e8b5ef705ba75 in virthost.ostest.test.metalkube.org:5000/localimages/local-release-image: manifest unknown]): registry.build05.ci.openshift.org/ci-op-wxyv2cs7/release@sha256:f457cec1b76f146536d0203d91d78777f7759022e6c665fe2c1e8b5ef705ba75: pinging container registry registry.build05.ci.openshift.org: Get "https://registry.build05.ci.openshift.org/v2/": dial tcp 3.212.211.54:443: connect: network is unreachable openshift-cluster-version 4m34s Warning Failed pod/version--sjlpf-kjs2v Failed to pull image "registry.build05.ci.openshift.org/ci-op-s5q5cy47/release@sha256:f457cec1b76f146536d0203d91d78777f7759022e6c665fe2c1e8b5ef705ba75": initializing source docker://registry.build05.ci.openshift.org/ci-op-s5q5cy47/release@sha256:f457cec1b76f146536d0203d91d78777f7759022e6c665fe2c1e8b5ef705ba75: (Mirrors also failed: [virthost.ostest.test.metalkube.org:5000/localimages/local-release-image@sha256:f457cec1b76f146536d0203d91d78777f7759022e6c665fe2c1e8b5ef705ba75: reading manifest sha256:f457cec1b76f146536d0203d91d78777f7759022e6c665fe2c1e8b5ef705ba75 in virthost.ostest.test.metalkube.org:5000/localimages/local-release-image: manifest unknown]... openshift-cluster-version 8m43s Warning Failed pod/version--vtsrh-kqmns Failed to pull image "registry.build05.ci.openshift.org/ci-op-vw43vp7q/release@sha256:e87cdacdf5c575ff99d4cca7ec38758512a408ac1653fef8dd7b2c4b85e295f4": initializing source docker://registry.build05.ci.openshift.org/ci-op-vw43vp7q/release@sha256:e87cdacdf5c575ff99d4cca7ec38758512a408ac1653fef8dd7b2c4b85e295f4: (Mirrors also failed: [virthost.ostest.test.metalkube.org:5000/localimages/local-release-image@sha256:e87cdacdf5c575ff99d4cca7ec38758512a408ac1653fef8dd7b2c4b85e295f4: reading manifest sha256:e87cdacdf5c575ff99d4cca7ec38758512a408ac1653fef8dd7b2c4b85e295f4 in virthost.ostest.test.metalkube.org:5000/localimages/local-release-image: manifest unknown]): registry.build05.ci.openshift.org/ci-op-vw43vp7q/release@sha256:e87cdacdf5c575ff99d4cca7ec38758512a408ac1653fef8dd7b2c4b85e295f4: pinging container registry registry.build05.ci.openshift.org: Get "https://registry.build05.ci.openshift.org/v2/": dial tcp 3.212.211.54:443: connect: network is unreachable
There is not much that we can do from CVO's points of view and I would like the bare-metal team and the test-platform team to check if there is something wrong with the infra structure such as the ephemeral cluster's networking and build05's registry. CVO has no platform specific code and the problem seems restricted to the Bare Metal environment. Based on the results from ci-search, "network is unreachable" feels like a thing in various jobs.
I set up the component as Bare Metal IPI because the test name is about Bare Metal. Please route the card to the right component if I am mistaken.
Version-Release number of selected component (if applicable):
4.18
How reproducible:
From sippy dashboard: Basis (historical) Release: 4.17 Success Rate: 99.53% Successes: 211 Failures: 1 Flakes: 0 Sample (being evaluated) Release: 4.18 Success Rate: 88.24% Successes: 45 Failures: 6 Flakes: 0
Steps to Reproduce:
1.
2.
3.
Actual results:
Expected results:
Additional info:
