-
Bug
-
Resolution: Not a Bug
-
Major
-
None
-
4.14
-
-
-
No
-
False
-
Description of problem:
Cluster operator image-registry is Degraded when using hive deploying OpenStack cluster. And the image-registry pod are CrashLoopBackOff. [hmx@fedora hive]$ oc get pod -n openshift-image-registry NAME READY STATUS RESTARTS AGE cluster-image-registry-operator-768f87dc77-dk84d 1/1 Running 1 (29m ago) 44m image-registry-5bb44d4b85-mt4dp 0/1 CrashLoopBackOff 10 (55s ago) 29m image-registry-5bb44d4b85-v5s2q 0/1 CrashLoopBackOff 10 (57s ago) 29m node-ca-47c5t 1/1 Running 0 29m node-ca-cxlzd 1/1 Running 0 29m node-ca-d7pn9 1/1 Running 0 27m node-ca-l54ct 1/1 Running 0 27m node-ca-rxmv9 1/1 Running 0 27m node-ca-zqgdb 1/1 Running 0 29m
Version-Release number of selected component (if applicable):
registry.ci.openshift.org/ocp/release:4.14.0-0.nightly-2024-04-04-065345
How reproducible:
Always
Steps to Reproduce:
1.Using hive deploying OpenStack cluster.
2.master/woeker nodes are running, but image-registry operator is Degraded.
3.
Actual results:
image-registry operator is Degraded.
Expected results:
image-registry operator is ready.
Additional info:
[hmx@fedora hive]$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.14.0-0.nightly-2024-04-04-065345 True False False 15m
baremetal 4.14.0-0.nightly-2024-04-04-065345 True False False 38m
cloud-controller-manager 4.14.0-0.nightly-2024-04-04-065345 True False False 41m
cloud-credential 4.14.0-0.nightly-2024-04-04-065345 True False False 43m
cluster-autoscaler 4.14.0-0.nightly-2024-04-04-065345 True False False 38m
config-operator 4.14.0-0.nightly-2024-04-04-065345 True False False 39m
console 4.14.0-0.nightly-2024-04-04-065345 True False False 20m
control-plane-machine-set 4.14.0-0.nightly-2024-04-04-065345 True False False 29m
csi-snapshot-controller 4.14.0-0.nightly-2024-04-04-065345 True False False 38m
dns 4.14.0-0.nightly-2024-04-04-065345 True False False 38m
etcd 4.14.0-0.nightly-2024-04-04-065345 True False False 31m
image-registry False True True 26m Available: The deployment does not have available replicas...
ingress 4.14.0-0.nightly-2024-04-04-065345 True False False 24m
insights 4.14.0-0.nightly-2024-04-04-065345 True False False 32m
kube-apiserver 4.14.0-0.nightly-2024-04-04-065345 True False False 26m
kube-controller-manager 4.14.0-0.nightly-2024-04-04-065345 True False False 35m
kube-scheduler 4.14.0-0.nightly-2024-04-04-065345 True False False 35m
kube-storage-version-migrator 4.14.0-0.nightly-2024-04-04-065345 True False False 38m
machine-api 4.14.0-0.nightly-2024-04-04-065345 True False False 23m
machine-approver 4.14.0-0.nightly-2024-04-04-065345 True False False 38m
machine-config 4.14.0-0.nightly-2024-04-04-065345 True False False 38m
marketplace 4.14.0-0.nightly-2024-04-04-065345 True False False 38m
monitoring 4.14.0-0.nightly-2024-04-04-065345 True False False 22m
network 4.14.0-0.nightly-2024-04-04-065345 True False False 40m
node-tuning 4.14.0-0.nightly-2024-04-04-065345 True False False 38m
openshift-apiserver 4.14.0-0.nightly-2024-04-04-065345 True False False 26m
openshift-controller-manager 4.14.0-0.nightly-2024-04-04-065345 True False False 34m
openshift-samples 4.14.0-0.nightly-2024-04-04-065345 True False False 25m
operator-lifecycle-manager 4.14.0-0.nightly-2024-04-04-065345 True False False 38m
operator-lifecycle-manager-catalog 4.14.0-0.nightly-2024-04-04-065345 True False False 38m
operator-lifecycle-manager-packageserver 4.14.0-0.nightly-2024-04-04-065345 True False False 26m
service-ca 4.14.0-0.nightly-2024-04-04-065345 True False False 39m
storage 4.14.0-0.nightly-2024-04-04-065345 True False False 37m
[hmx@fedora hive]$ oc get machineset -A
NAMESPACE NAME DESIRED CURRENT READY AVAILABLE AGE
openshift-machine-api mihuangoshv-ww9wt-worker-0 3 3 3 3 45m
[hmx@fedora hive]$
[hmx@fedora hive]$ oc get machines -A
NAMESPACE NAME PHASE TYPE REGION ZONE AGE
openshift-machine-api mihuangoshv-ww9wt-master-0 Running m1.xlarge.disk regionOne nova 45m
openshift-machine-api mihuangoshv-ww9wt-master-1 Running m1.xlarge.disk regionOne nova 45m
openshift-machine-api mihuangoshv-ww9wt-master-2 Running m1.xlarge.disk regionOne nova 45m
openshift-machine-api mihuangoshv-ww9wt-worker-0-bzd9t Running m1.xlarge.disk regionOne nova 38m
openshift-machine-api mihuangoshv-ww9wt-worker-0-g4qqv Running m1.xlarge.disk regionOne nova 38m
openshift-machine-api mihuangoshv-ww9wt-worker-0-l9rn9 Running m1.xlarge.disk regionOne nova 38m
[hmx@fedora hive]$ oc get pod -n openshift-image-registry
NAME READY STATUS RESTARTS AGE
cluster-image-registry-operator-768f87dc77-dk84d 1/1 Running 1 (29m ago) 44m
image-registry-5bb44d4b85-mt4dp 0/1 CrashLoopBackOff 10 (55s ago) 29m
image-registry-5bb44d4b85-v5s2q 0/1 CrashLoopBackOff 10 (57s ago) 29m
node-ca-47c5t 1/1 Running 0 29m
node-ca-cxlzd 1/1 Running 0 29m
node-ca-d7pn9 1/1 Running 0 27m
node-ca-l54ct 1/1 Running 0 27m
node-ca-rxmv9 1/1 Running 0 27m
node-ca-zqgdb 1/1 Running 0 29m
[hmx@fedora hive]$ oc describe pod image-registry-5bb44d4b85-mt4dp -o yaml
error: unknown shorthand flag: 'o' in -o
See 'oc describe --help' for usage.
[hmx@fedora hive]$ oc describe pod image-registry-5bb44d4b85-mt4dp
Error from server (NotFound): pods "image-registry-5bb44d4b85-mt4dp" not found
[hmx@fedora hive]$ oc describe pod image-registry-5bb44d4b85-mt4dp -n openshift-image-registry
Name: image-registry-5bb44d4b85-mt4dp
Namespace: openshift-image-registry
Priority: 2000000000
Priority Class Name: system-cluster-critical
Service Account: registry
Node: mihuangoshv-ww9wt-worker-0-l9rn9/192.168.2.192
Start Time: Mon, 08 Apr 2024 20:05:11 +0800
Labels: docker-registry=default
pod-template-hash=5bb44d4b85
Annotations: imageregistry.operator.openshift.io/dependencies-checksum: sha256:0b0dea928daf2900e7ef2411c4b4a4191262b8545b0d29d4a24b7e8744b65b7c
k8s.ovn.org/pod-networks:
{"default":{"ip_addresses":["10.131.0.8/23"],"mac_address":"0a:58:0a:83:00:08","gateway_ips":["10.131.0.1"],"routes":[{"dest":"10.128.0.0/...
k8s.v1.cni.cncf.io/network-status:
[{
"name": "ovn-kubernetes",
"interface": "eth0",
"ips": [
"10.131.0.8"
],
"mac": "0a:58:0a:83:00:08",
"default": true,
"dns": {}
}]
openshift.io/scc: restricted-v2
seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status: Running
SeccompProfile: RuntimeDefault
IP: 10.131.0.8
IPs:
IP: 10.131.0.8
Controlled By: ReplicaSet/image-registry-5bb44d4b85
Containers:
registry:
Container ID: cri-o://34e8b0fe34e753aa2aa034fc4b9358019a6222b2a4d7dd38060855d83462e342
Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc653a9c24392d7e51552bdfef8e3eb681eea1b04a1cc8cbd11069b4e19043d5
Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc653a9c24392d7e51552bdfef8e3eb681eea1b04a1cc8cbd11069b4e19043d5
Port: 5000/TCP
Host Port: 0/TCP
Command:
/bin/sh
-c
mkdir -p /etc/pki/ca-trust/extracted/edk2 /etc/pki/ca-trust/extracted/java /etc/pki/ca-trust/extracted/openssl /etc/pki/ca-trust/extracted/pem && update-ca-trust extract && exec /usr/bin/dockerregistry
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Mon, 08 Apr 2024 20:31:27 +0800
Finished: Mon, 08 Apr 2024 20:31:27 +0800
Ready: False
Restart Count: 10
Requests:
cpu: 100m
memory: 256Mi
Liveness: http-get https://:5000/healthz delay=5s timeout=5s period=10s #success=1 #failure=3
Readiness: http-get https://:5000/healthz delay=15s timeout=5s period=10s #success=1 #failure=3
Environment:
REGISTRY_STORAGE: swift
REGISTRY_STORAGE_SWIFT_CONTAINER: mihuangoshv-ww9wt-image-registry-giyxpcgenqsafvnmyeljrrwdwkglp
REGISTRY_STORAGE_SWIFT_AUTHURL: https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13000/v3/auth/tokens
REGISTRY_STORAGE_SWIFT_USERNAME: <set to the key 'REGISTRY_STORAGE_SWIFT_USERNAME' in secret 'image-registry-private-configuration'> Optional: false
REGISTRY_STORAGE_SWIFT_PASSWORD: <set to the key 'REGISTRY_STORAGE_SWIFT_PASSWORD' in secret 'image-registry-private-configuration'> Optional: false
REGISTRY_STORAGE_SWIFT_APPLICATIONCREDENTIALID: <set to the key 'REGISTRY_STORAGE_SWIFT_APPLICATIONCREDENTIALID' in secret 'image-registry-private-configuration'> Optional: false
REGISTRY_STORAGE_SWIFT_APPLICATIONCREDENTIALNAME: <set to the key 'REGISTRY_STORAGE_SWIFT_APPLICATIONCREDENTIALNAME' in secret 'image-registry-private-configuration'> Optional: false
REGISTRY_STORAGE_SWIFT_APPLICATIONCREDENTIALSECRET: <set to the key 'REGISTRY_STORAGE_SWIFT_APPLICATIONCREDENTIALSECRET' in secret 'image-registry-private-configuration'> Optional: false
REGISTRY_STORAGE_SWIFT_AUTHVERSION: 3
REGISTRY_STORAGE_SWIFT_DOMAIN: redhat.com
REGISTRY_STORAGE_SWIFT_TENANT: openshift-qe-jenkins
REGISTRY_STORAGE_SWIFT_TENANTID: 542c6ebd48bf40fa857fc245c7572e30
REGISTRY_STORAGE_SWIFT_REGION: regionOne
REGISTRY_HTTP_ADDR: :5000
REGISTRY_HTTP_NET: tcp
REGISTRY_HTTP_SECRET: 24ee7e46142fdb502750944fb8c59e9850b9e3faf3c3f8bfd558a2c80ed5b017b595147cf3856fc38db2f3d474f74797a340a1a13876446c027362c691f97307
REGISTRY_LOG_LEVEL: info
REGISTRY_OPENSHIFT_QUOTA_ENABLED: true
REGISTRY_STORAGE_CACHE_BLOBDESCRIPTOR: inmemory
REGISTRY_STORAGE_DELETE_ENABLED: true
REGISTRY_HEALTH_STORAGEDRIVER_ENABLED: true
REGISTRY_HEALTH_STORAGEDRIVER_INTERVAL: 10s
REGISTRY_HEALTH_STORAGEDRIVER_THRESHOLD: 1
REGISTRY_OPENSHIFT_METRICS_ENABLED: true
REGISTRY_OPENSHIFT_SERVER_ADDR: image-registry.openshift-image-registry.svc:5000
REGISTRY_HTTP_TLS_CERTIFICATE: /etc/secrets/tls.crt
REGISTRY_HTTP_TLS_KEY: /etc/secrets/tls.key
Mounts:
/etc/pki/ca-trust/extracted from ca-trust-extracted (rw)
/etc/pki/ca-trust/source/anchors from registry-certificates (rw)
/etc/secrets from registry-tls (rw)
/usr/share/pki/ca-trust-source from trusted-ca (rw)
/var/lib/kubelet/ from installation-pull-secrets (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pctgm (ro)
/var/run/secrets/openshift/serviceaccount from bound-sa-token (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
registry-tls:
Type: Projected (a volume that contains injected data from multiple sources)
SecretName: image-registry-tls
SecretOptionalName: <nil>
ca-trust-extracted:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
registry-certificates:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: image-registry-certificates
Optional: false
trusted-ca:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: trusted-ca
Optional: true
installation-pull-secrets:
Type: Secret (a volume populated by a Secret)
SecretName: installation-pull-secrets
Optional: true
bound-sa-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3600
kube-api-access-pctgm:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
ConfigMapName: openshift-service-ca.crt
ConfigMapOptional: <nil>
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints: kubernetes.io/hostname:DoNotSchedule when max skew 1 is exceeded for selector docker-registry=default
node-role.kubernetes.io/worker:DoNotSchedule when max skew 1 is exceeded for selector docker-registry=default
topology.kubernetes.io/zone:DoNotSchedule when max skew 1 is exceeded for selector docker-registry=default
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 29m default-scheduler 0/3 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..
Warning FailedScheduling 27m (x2 over 29m) default-scheduler 0/3 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..
Normal Scheduled 27m default-scheduler Successfully assigned openshift-image-registry/image-registry-5bb44d4b85-mt4dp to mihuangoshv-ww9wt-worker-0-l9rn9
Normal AddedInterface 27m multus Add eth0 [10.131.0.8/23] from ovn-kubernetes
Normal Started 27m (x4 over 27m) kubelet Started container registry
Normal Pulled 26m (x5 over 27m) kubelet Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc653a9c24392d7e51552bdfef8e3eb681eea1b04a1cc8cbd11069b4e19043d5" already present on machine
Normal Created 26m (x5 over 27m) kubelet Created container registry
Warning BackOff 2m47s (x129 over 27m) kubelet Back-off restarting failed container registry in pod image-registry-5bb44d4b85-mt4dp_openshift-image-registry(ad37eedf-7dc7-4d8c-9527-c4db613ddac3)
