-
Bug
-
Resolution: Unresolved
-
Critical
-
4.16
-
None
-
Critical
-
No
-
Proposed
-
False
-
-
Kubernetes 1.29 changed the way the CCM assigns IP addresses. Also, changed the --bind-address to localhost. This fixes those issues.
-
Bug Fix
-
In Progress
In PowerVS, when I try and deploy a 4.16 cluster, I see the following:
Description of problem:
[inner hamzy@li-3d08e84c-2e1c-11b2-a85c-e2db7bb078fc hamzy-release]$ oc get pods -n openshift-cloud-controller-manager NAME READY STATUS RESTARTS AGE powervs-cloud-controller-manager-6b6fbcc9db-9rhtj 0/1 CrashLoopBackOff 4 (10s ago) 2m47s powervs-cloud-controller-manager-6b6fbcc9db-wnvck 0/1 CrashLoopBackOff 3 (49s ago) 2m46s [inner hamzy@li-3d08e84c-2e1c-11b2-a85c-e2db7bb078fc hamzy-release]$ oc logs pod/powervs-cloud-controller-manager-6b6fbcc9db-9rhtj -n openshift-cloud-controller-manager Error from server: no preferred addresses found; known addresses: [] [inner hamzy@li-3d08e84c-2e1c-11b2-a85c-e2db7bb078fc hamzy-release]$ oc logs pod/powervs-cloud-controller-manager-6b6fbcc9db-wnvck -n openshift-cloud-controller-manager Error from server: no preferred addresses found; known addresses: []
Version-Release number of selected component (if applicable):
4.16.0-0.nightly-ppc64le-2024-01-07-111144
How reproducible:
Aways
Steps to Reproduce:
1. Deploy OpenShift cluster
On the master-0 node, I see:
[core@rdr-hamzy-test-wdc06-fs5m2-master-0 ~]$ sudo crictl ps -a CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD a048556553827 ec3035a371e09312254a277d5eb9affba2930adbd4018f7557899a2f3d76bc88 18 seconds ago Exited kube-rbac-proxy 7 0381a589d57cd cluster-cloud-controller-manager-operator-94dd5b468-kxqw5 a326f7ec83ddb 60f5c9455518c79a9797cfbeab0b3530dae1bf77554eccc382ff12d99053efd1 11 minutes ago Running config-sync-controllers 0 0381a589d57cd cluster-cloud-controller-manager-operator-94dd5b468-kxqw5 ddaa6999b5b86 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:60eff87ed56ee4761fd55caa4712e6bea47dccaa11c59ba53a6d5697eacc7d32 11 minutes ago Running cluster-cloud-controller-manager 0 0381a589d57cd cluster-cloud-controller-manager-operator-94dd5b468-kxqw5
The failing pod has this as its log:
[core@rdr-hamzy-test-wdc06-fs5m2-master-0 ~]$ sudo crictl logs a048556553827 Flag --logtostderr has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components I0108 18:09:12.320332 1 flags.go:64] FLAG: --add-dir-header="false" I0108 18:09:12.320401 1 flags.go:64] FLAG: --allow-paths="[]" I0108 18:09:12.320413 1 flags.go:64] FLAG: --alsologtostderr="false" I0108 18:09:12.320420 1 flags.go:64] FLAG: --auth-header-fields-enabled="false" I0108 18:09:12.320427 1 flags.go:64] FLAG: --auth-header-groups-field-name="x-remote-groups" I0108 18:09:12.320435 1 flags.go:64] FLAG: --auth-header-groups-field-separator="|" I0108 18:09:12.320441 1 flags.go:64] FLAG: --auth-header-user-field-name="x-remote-user" I0108 18:09:12.320447 1 flags.go:64] FLAG: --auth-token-audiences="[]" I0108 18:09:12.320454 1 flags.go:64] FLAG: --client-ca-file="" I0108 18:09:12.320460 1 flags.go:64] FLAG: --config-file="/etc/kube-rbac-proxy/config-file.yaml" I0108 18:09:12.320467 1 flags.go:64] FLAG: --help="false" I0108 18:09:12.320473 1 flags.go:64] FLAG: --http2-disable="false" I0108 18:09:12.320479 1 flags.go:64] FLAG: --http2-max-concurrent-streams="100" I0108 18:09:12.320486 1 flags.go:64] FLAG: --http2-max-size="262144" I0108 18:09:12.320492 1 flags.go:64] FLAG: --ignore-paths="[]" I0108 18:09:12.320500 1 flags.go:64] FLAG: --insecure-listen-address="" I0108 18:09:12.320506 1 flags.go:64] FLAG: --kubeconfig="" I0108 18:09:12.320512 1 flags.go:64] FLAG: --log-backtrace-at=":0" I0108 18:09:12.320520 1 flags.go:64] FLAG: --log-dir="" I0108 18:09:12.320526 1 flags.go:64] FLAG: --log-file="" I0108 18:09:12.320531 1 flags.go:64] FLAG: --log-file-max-size="1800" I0108 18:09:12.320537 1 flags.go:64] FLAG: --log-flush-frequency="5s" I0108 18:09:12.320543 1 flags.go:64] FLAG: --logtostderr="true" I0108 18:09:12.320550 1 flags.go:64] FLAG: --oidc-ca-file="" I0108 18:09:12.320556 1 flags.go:64] FLAG: --oidc-clientID="" I0108 18:09:12.320564 1 flags.go:64] FLAG: --oidc-groups-claim="groups" I0108 18:09:12.320570 1 flags.go:64] FLAG: --oidc-groups-prefix="" I0108 18:09:12.320576 1 flags.go:64] FLAG: --oidc-issuer="" I0108 18:09:12.320581 1 flags.go:64] FLAG: --oidc-sign-alg="[RS256]" I0108 18:09:12.320590 1 flags.go:64] FLAG: --oidc-username-claim="email" I0108 18:09:12.320595 1 flags.go:64] FLAG: --one-output="false" I0108 18:09:12.320601 1 flags.go:64] FLAG: --proxy-endpoints-port="0" I0108 18:09:12.320608 1 flags.go:64] FLAG: --secure-listen-address="0.0.0.0:9258" I0108 18:09:12.320614 1 flags.go:64] FLAG: --skip-headers="false" I0108 18:09:12.320620 1 flags.go:64] FLAG: --skip-log-headers="false" I0108 18:09:12.320626 1 flags.go:64] FLAG: --stderrthreshold="2" I0108 18:09:12.320631 1 flags.go:64] FLAG: --tls-cert-file="/etc/tls/private/tls.crt" I0108 18:09:12.320637 1 flags.go:64] FLAG: --tls-cipher-suites="[TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305]" I0108 18:09:12.320654 1 flags.go:64] FLAG: --tls-min-version="VersionTLS12" I0108 18:09:12.320661 1 flags.go:64] FLAG: --tls-private-key-file="/etc/tls/private/tls.key" I0108 18:09:12.320667 1 flags.go:64] FLAG: --tls-reload-interval="1m0s" I0108 18:09:12.320674 1 flags.go:64] FLAG: --upstream="http://127.0.0.1:9257/" I0108 18:09:12.320681 1 flags.go:64] FLAG: --upstream-ca-file="" I0108 18:09:12.320686 1 flags.go:64] FLAG: --upstream-client-cert-file="" I0108 18:09:12.320692 1 flags.go:64] FLAG: --upstream-client-key-file="" I0108 18:09:12.320697 1 flags.go:64] FLAG: --upstream-force-h2c="false" I0108 18:09:12.320703 1 flags.go:64] FLAG: --v="3" I0108 18:09:12.320709 1 flags.go:64] FLAG: --version="false" I0108 18:09:12.320719 1 flags.go:64] FLAG: --vmodule="" I0108 18:09:12.320735 1 kube-rbac-proxy.go:578] Reading config file: /etc/kube-rbac-proxy/config-file.yaml I0108 18:09:12.321427 1 kube-rbac-proxy.go:285] Valid token audiences: I0108 18:09:12.321473 1 kube-rbac-proxy.go:399] Reading certificate files E0108 18:09:12.321519 1 run.go:74] "command failed" err="failed to initialize certificate reloader: error loading certificates: error loading certificate: open /etc/tls/private/tls.crt: no such file or directory"
When I describe the pod, I see:
[inner hamzy@li-3d08e84c-2e1c-11b2-a85c-e2db7bb078fc hamzy-release]$ oc describe pod/powervs-cloud-controller-manager-6b6fbcc9db-9rhtj -n openshift-cloud-controller-manager
Name: powervs-cloud-controller-manager-6b6fbcc9db-9rhtj
Namespace: openshift-cloud-controller-manager
Priority: 2000000000
Priority Class Name: system-cluster-critical
Service Account: cloud-controller-manager
Node: rdr-hamzy-test-wdc06-fs5m2-master-2/
Start Time: Mon, 08 Jan 2024 11:57:45 -0600
Labels: infrastructure.openshift.io/cloud-controller-manager=PowerVS
k8s-app=powervs-cloud-controller-manager
pod-template-hash=6b6fbcc9db
Annotations: operator.openshift.io/config-hash: 09205e81b4dc20086c29ddbdd3fccc29a675be94b2779756a0e748dd9ba91e40
Status: Running
IP:
IPs: <none>
Controlled By: ReplicaSet/powervs-cloud-controller-manager-6b6fbcc9db
Containers:
cloud-controller-manager:
Container ID: cri-o://4365a326d05ecaac8e4114efabb4a46e01a308459ad30438d742b4829c24a717
Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dd2cf78ddeed971d38731d27ce293501547b960cefc3aadaa220186eded8a09
Image ID: 65401afa73528f9a425a9d7f5dee8a9de8d9d3d82c8fd84cd653b16409093836
Port: 10258/TCP
Host Port: 10258/TCP
Command:
/bin/bash
-c
#!/bin/bash
set -o allexport
if [[ -f /etc/kubernetes/apiserver-url.env ]]; then
source /etc/kubernetes/apiserver-url.env
fi
exec /bin/ibm-cloud-controller-manager \
--bind-address=$(POD_IP_ADDRESS) \
--use-service-account-credentials=true \
--configure-cloud-routes=false \
--cloud-provider=ibm \
--cloud-config=/etc/ibm/cloud.conf \
--profiling=false \
--leader-elect=true \
--leader-elect-lease-duration=137s \
--leader-elect-renew-deadline=107s \
--leader-elect-retry-period=26s \
--leader-elect-resource-namespace=openshift-cloud-controller-manager \
--tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_AES_128_GCM_SHA256,TLS_CHACHA20_POLY1305_SHA256,TLS_AES_256_GCM_SHA384 \
--v=2
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 08 Jan 2024 12:35:12 -0600
Finished: Mon, 08 Jan 2024 12:35:12 -0600
Ready: False
Restart Count: 12
Requests:
cpu: 75m
memory: 60Mi
Liveness: http-get https://:10258/healthz delay=300s timeout=160s period=10s #success=1 #failure=3
Environment:
POD_IP_ADDRESS: (v1:status.podIP)
VPCCTL_CLOUD_CONFIG: /etc/ibm/cloud.conf
ENABLE_VPC_PUBLIC_ENDPOINT: true
Mounts:
/etc/ibm from cloud-conf (rw)
/etc/kubernetes from host-etc-kube (ro)
/etc/pki/ca-trust/extracted/pem from trusted-ca (ro)
/etc/vpc from ibm-cloud-credentials (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z5xdm (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
trusted-ca:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: ccm-trusted-ca
Optional: false
host-etc-kube:
Type: HostPath (bare host directory volume)
Path: /etc/kubernetes
HostPathType: Directory
cloud-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: cloud-conf
Optional: false
ibm-cloud-credentials:
Type: Secret (a volume populated by a Secret)
SecretName: ibm-cloud-credentials
Optional: false
kube-api-access-z5xdm:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
ConfigMapName: openshift-service-ca.crt
ConfigMapOptional: <nil>
QoS Class: Burstable
Node-Selectors: node-role.kubernetes.io/master=
Tolerations: node-role.kubernetes.io/master:NoSchedule op=Exists
node.cloudprovider.kubernetes.io/uninitialized:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 120s
node.kubernetes.io/not-ready:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists for 120s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 38m default-scheduler Successfully assigned openshift-cloud-controller-manager/powervs-cloud-controller-manager-6b6fbcc9db-9rhtj to rdr-hamzy-test-wdc06-fs5m2-master-2
Normal Pulling 38m kubelet Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dd2cf78ddeed971d38731d27ce293501547b960cefc3aadaa220186eded8a09"
Normal Pulled 37m kubelet Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dd2cf78ddeed971d38731d27ce293501547b960cefc3aadaa220186eded8a09" in 36.694s (36.694s including waiting)
Normal Started 36m (x4 over 37m) kubelet Started container cloud-controller-manager
Normal Created 35m (x5 over 37m) kubelet Created container cloud-controller-manager
Normal Pulled 35m (x4 over 37m) kubelet Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dd2cf78ddeed971d38731d27ce293501547b960cefc3aadaa220186eded8a09" already present on machine
Warning BackOff 2m57s (x166 over 37m) kubelet Back-off restarting failed container cloud-controller-manager in pod powervs-cloud-controller-manager-6b6fbcc9db-9rhtj_openshift-cloud-controller-manager(bf58b824-b1a2-4d2e-8735-22723642a24a)
- links to
-
RHEA-2024:0041
OpenShift Container Platform 4.16.z bug fix update
