-
Bug
-
Resolution: Done-Errata
-
Critical
-
4.15, 4.16
Description of problem:
Install IPI cluster against 4.15 nightly build on Azure MAG and Azure Stack Hub or with Azure workload identity, image-registry co is degraded with different errors.
On MAG:
$ oc get co image-registry
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
image-registry 4.15.0-0.nightly-2024-02-16-235514 True False True 5h44m AzurePathFixControllerDegraded: Migration failed: panic: Get "https://imageregistryjima41xvvww.blob.core.windows.net/jima415a-hfxfh-image-registry-vbibdmawmsvqckhvmmiwisebryohfbtm?comp=list&prefix=docker&restype=container": dial tcp: lookup imageregistryjima41xvvww.blob.core.windows.net on 172.30.0.10:53: no such host...
$ oc get pod -n openshift-image-registry
NAME READY STATUS RESTARTS AGE
azure-path-fix-ssn5w 0/1 Error 0 5h47m
cluster-image-registry-operator-86cdf775c7-7brn6 1/1 Running 1 (5h50m ago) 5h58m
image-registry-5c6796b86d-46lvx 1/1 Running 0 5h47m
image-registry-5c6796b86d-9st5d 1/1 Running 0 5h47m
node-ca-48lsh 1/1 Running 0 5h44m
node-ca-5rrsl 1/1 Running 0 5h47m
node-ca-8sc92 1/1 Running 0 5h47m
node-ca-h6trz 1/1 Running 0 5h47m
node-ca-hm7s2 1/1 Running 0 5h47m
node-ca-z7tv8 1/1 Running 0 5h44m
$ oc logs azure-path-fix-ssn5w -n openshift-image-registry
panic: Get "https://imageregistryjima41xvvww.blob.core.windows.net/jima415a-hfxfh-image-registry-vbibdmawmsvqckhvmmiwisebryohfbtm?comp=list&prefix=docker&restype=container": dial tcp: lookup imageregistryjima41xvvww.blob.core.windows.net on 172.30.0.10:53: no such hostgoroutine 1 [running]:
main.main()
/go/src/github.com/openshift/cluster-image-registry-operator/cmd/move-blobs/main.go:49 +0x125
The blob storage endpoint seems not correct, should be:
$ az storage account show -n imageregistryjima41xvvww -g jima415a-hfxfh-rg --query primaryEndpoints
{
"blob": "https://imageregistryjima41xvvww.blob.core.usgovcloudapi.net/",
"dfs": "https://imageregistryjima41xvvww.dfs.core.usgovcloudapi.net/",
"file": "https://imageregistryjima41xvvww.file.core.usgovcloudapi.net/",
"internetEndpoints": null,
"microsoftEndpoints": null,
"queue": "https://imageregistryjima41xvvww.queue.core.usgovcloudapi.net/",
"table": "https://imageregistryjima41xvvww.table.core.usgovcloudapi.net/",
"web": "https://imageregistryjima41xvvww.z2.web.core.usgovcloudapi.net/"
}
On Azure Stack Hub:
$ oc get co image-registry
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
image-registry 4.15.0-0.nightly-2024-02-16-235514 True False True 3h32m AzurePathFixControllerDegraded: Migration failed: panic: open : no such file or directory...
$ oc get pod -n openshift-image-registry
NAME READY STATUS RESTARTS AGE
azure-path-fix-8jdg7 0/1 Error 0 3h35m
cluster-image-registry-operator-86cdf775c7-jwnd4 1/1 Running 1 (3h38m ago) 3h54m
image-registry-658669fbb4-llv8z 1/1 Running 0 3h35m
image-registry-658669fbb4-lmfr6 1/1 Running 0 3h35m
node-ca-2jkjx 1/1 Running 0 3h35m
node-ca-dcg2v 1/1 Running 0 3h35m
node-ca-q6xmn 1/1 Running 0 3h35m
node-ca-r46r2 1/1 Running 0 3h35m
node-ca-s8jkb 1/1 Running 0 3h35m
node-ca-ww6ql 1/1 Running 0 3h35m
$ oc logs azure-path-fix-8jdg7 -n openshift-image-registry
panic: open : no such file or directorygoroutine 1 [running]:
main.main()
/go/src/github.com/openshift/cluster-image-registry-operator/cmd/move-blobs/main.go:36 +0x145
On cluster with Azure workload identity:
Some operator's PROGRESSING is True
image-registry 4.15.0-0.nightly-2024-02-16-235514 True True False 43m Progressing: The deployment has not completed...
pod azure-path-fix is in CreateContainerConfigError status, and get error in its Event.
"state": {
"waiting": {
"message": "couldn't find key REGISTRY_STORAGE_AZURE_ACCOUNTKEY in Secret openshift-image-registry/image-registry-private-configuration",
"reason": "CreateContainerConfigError"
}
}
Version-Release number of selected component (if applicable):
4.15.0-0.nightly-2024-02-16-235514
How reproducible:
Always
Steps to Reproduce:
1. Install IPI cluster on MAG or Azure Stack Hub or config Azure workload identity
2.
3.
Actual results:
Installation failed and image-registry operator is degraded
Expected results:
Installation is successful.
Additional info:
Seems that issue is related with https://github.com/openshift/image-registry/pull/393
- blocks
-
OCPBUGS-29755 image-registry co is degraded on Azure MAG, Azure Stack Hub cloud or with azure workload identity
-
- Closed
-
- clones
-
OCPBUGS-29637 image-registry co is degraded on Azure MAG, Azure Stack Hub cloud or with azure workload identity
-
- Closed
-
- depends on
-
OCPBUGS-29637 image-registry co is degraded on Azure MAG, Azure Stack Hub cloud or with azure workload identity
-
- Closed
-
- is cloned by
-
OCPBUGS-29755 image-registry co is degraded on Azure MAG, Azure Stack Hub cloud or with azure workload identity
-
- Closed
-
- links to
-
RHSA-2023:7198
OpenShift Container Platform 4.15 security update