-
Bug
-
Resolution: Not a Bug
-
Major
-
None
-
4.11
-
Important
-
Rejected
-
False
-
-
Customer Escalated
-
Description of problem:
During OpenShift Container Platform 4.10 to 4.11 specific pod objects are not re-created as expected, leaving them with openshift.io/scc: restricted applied and causing undesired effects when trying to append annotion for example. This does impact AMQ Streams for example, as the Operator does enforce a specific order when restartind pods and implements this using annotion. Due to this behavior/issue, the AMQ Operator is failing to apply the necessary annotation and therefore upgrade to OpenShift Container Platform 4.11 for customers using AMQ Streams is currently blocked.
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.10.41 True False 42h Cluster version is 4.10.41
$ oc get pod
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 6/6 Running 12 43h
alertmanager-main-1 6/6 Running 0 43h
cluster-monitoring-operator-5b46cc878c-wplnz 2/2 Running 0 44h
grafana-c8fdffc9-bpsp7 3/3 Running 0 43h
kube-state-metrics-77f55bd599-vh6wv 3/3 Running 0 43h
node-exporter-45px7 2/2 Running 0 43h
node-exporter-gwgwv 2/2 Running 0 43h
node-exporter-hwlzb 2/2 Running 0 43h
node-exporter-j5bgg 2/2 Running 0 43h
node-exporter-k2b9g 2/2 Running 4 43h
node-exporter-vptzw 2/2 Running 0 43h
openshift-state-metrics-77c76b46d8-8rvmx 3/3 Running 0 43h
prometheus-adapter-856f9bf45d-l8tvl 1/1 Running 0 20h
prometheus-adapter-856f9bf45d-zpkkf 1/1 Running 0 20h
prometheus-k8s-0 6/6 Running 12 42h
prometheus-k8s-1 6/6 Running 0 42h
prometheus-operator-5685df9747-vm674 2/2 Running 0 43h
sre-dns-latency-exporter-cdrkb 1/1 Running 0 22h
sre-dns-latency-exporter-lgl94 1/1 Running 0 22h
sre-dns-latency-exporter-mrmpw 1/1 Running 0 22h
sre-dns-latency-exporter-rgmvg 1/1 Running 0 22h
sre-dns-latency-exporter-tmgcn 1/1 Running 0 22h
sre-dns-latency-exporter-zplqt 1/1 Running 0 22h
telemeter-client-7d4c49494c-z2kfm 3/3 Running 0 43h
thanos-querier-595bbd5b84-jw5r7 6/6 Running 0 43h
thanos-querier-595bbd5b84-rlp5h 6/6 Running 12 43h
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.11.16 True False 2m27s Cluster version is 4.11.16
$ oc get pod
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 6/6 Running 0 14m
alertmanager-main-1 6/6 Running 0 11m
cluster-monitoring-operator-7788945d44-kt9c2 2/2 Running 0 8m16s
kube-state-metrics-68667ff848-ddlbq 3/3 Running 0 14m
node-exporter-bmc7m 2/2 Running 2 36m
node-exporter-bswtq 2/2 Running 2 36m
node-exporter-d2z56 2/2 Running 2 36m
node-exporter-d9bk2 2/2 Running 2 36m
node-exporter-v6nxv 2/2 Running 2 36m
node-exporter-vdfrz 2/2 Running 2 35m
openshift-state-metrics-6959bf55f6-8lm5n 3/3 Running 0 14m
prometheus-adapter-5f8878547b-8lsk4 1/1 Running 0 11m
prometheus-adapter-5f8878547b-fb2rq 1/1 Running 0 14m
prometheus-k8s-0 6/6 Running 0 14m
prometheus-k8s-1 6/6 Running 0 11m
prometheus-operator-admission-webhook-6c69dfd44-jgv7b 1/1 Running 0 14m
prometheus-operator-admission-webhook-6c69dfd44-xz6sf 1/1 Running 0 11m
prometheus-operator-fd67c7b8f-zb4lz 2/2 Running 0 8m17s
sre-dns-latency-exporter-cdrkb 1/1 Running 1 23h
sre-dns-latency-exporter-lgl94 1/1 Running 1 23h
sre-dns-latency-exporter-mrmpw 1/1 Running 1 23h
sre-dns-latency-exporter-rgmvg 1/1 Running 1 23h
sre-dns-latency-exporter-tmgcn 1/1 Running 1 23h
sre-dns-latency-exporter-zplqt 1/1 Running 1 23h
telemeter-client-5fd48c7bbb-vhcxl 3/3 Running 0 14m
thanos-querier-5f6584567f-2b2t2 6/6 Running 0 14m
thanos-querier-5f6584567f-c6p7s 6/6 Running 0 11m
$ oc get pod sre-dns-latency-exporter-cdrkb -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
[...]
openshift.io/scc: restricted
$ oc annotate pod -n openshift-monitoring sre-dns-latency-exporter-cdrkb "test=foo"
Error from server (Forbidden): pods "sre-dns-latency-exporter-cdrkb" is forbidden: unable to validate against any security context constraint: [spec.volumes[0]: Invalid value: "configMap": configMap volumes are not allowed to be used, spec.volumes[1]: Invalid value: "projected": projected volumes are not allowed to be used]
$ oc get pod
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 6/6 Running 0 21m
alertmanager-main-1 6/6 Running 0 18m
cluster-monitoring-operator-7788945d44-kt9c2 2/2 Running 0 15m
kube-state-metrics-68667ff848-ddlbq 3/3 Running 0 21m
node-exporter-bmc7m 2/2 Running 2 43m
node-exporter-bswtq 2/2 Running 2 43m
node-exporter-d2z56 2/2 Running 2 43m
node-exporter-d9bk2 2/2 Running 2 43m
node-exporter-v6nxv 2/2 Running 2 43m
node-exporter-vdfrz 2/2 Running 2 42m
openshift-state-metrics-6959bf55f6-8lm5n 3/3 Running 0 21m
prometheus-adapter-5f8878547b-8lsk4 1/1 Running 0 18m
prometheus-adapter-5f8878547b-fb2rq 1/1 Running 0 21m
prometheus-k8s-0 6/6 Running 0 21m
prometheus-k8s-1 6/6 Running 0 18m
prometheus-operator-admission-webhook-6c69dfd44-jgv7b 1/1 Running 0 21m
prometheus-operator-admission-webhook-6c69dfd44-xz6sf 1/1 Running 0 18m
prometheus-operator-fd67c7b8f-zb4lz 2/2 Running 0 15m
sre-dns-latency-exporter-cdrkb 1/1 Running 1 23h
sre-dns-latency-exporter-dkvxs 1/1 Running 0 18s
sre-dns-latency-exporter-mrmpw 1/1 Running 1 23h
sre-dns-latency-exporter-rgmvg 1/1 Running 1 23h
sre-dns-latency-exporter-tmgcn 1/1 Running 1 23h
sre-dns-latency-exporter-zplqt 1/1 Running 1 23h
telemeter-client-5fd48c7bbb-vhcxl 3/3 Running 0 21m
thanos-querier-5f6584567f-2b2t2 6/6 Running 0 21m
thanos-querier-5f6584567f-c6p7s 6/6 Running 0 18m
$ oc get pod sre-dns-latency-exporter-dkvxs -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
[...]
openshift.io/scc: restricted-v2
$ oc annotate pod -n openshift-monitoring sre-dns-latency-exporter-dkvxs "test=foo"
pod/sre-dns-latency-exporter-dkvxs annotated
Version-Release number of selected component (if applicable):
- OpenShift Container Platform 4.10 to 4.11 update
How reproducible:
- Always
Steps to Reproduce:
1. Install OpenShift Container Platform 4.10 2. Apply the DaemonSet from https://github.com/sreber84/dns-latency 3. Update to OpenShift Container Platform 4.11 4. Run oc annotate pod -n openshift-monitoring sre-dns-latency-exporter-<id> "test=foo"
Actual results:
$ oc annotate pod -n openshift-monitoring sre-dns-latency-exporter-cdrkb "test=foo"
Error from server (Forbidden): pods "sre-dns-latency-exporter-cdrkb" is forbidden: unable to validate against any security context constraint: [spec.volumes[0]: Invalid value: "configMap": configMap volumes are not allowed to be used, spec.volumes[1]: Invalid value: "projected": projected volumes are not allowed to be used]
$ oc get pod sre-dns-latency-exporter-cdrkb -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
[...]
openshift.io/scc: restricted
Expected results:
$ oc annotate pod -n openshift-monitoring sre-dns-latency-exporter-dkvxs "test=foo"
pod/sre-dns-latency-exporter-dkvxs annotated
$ oc get pod sre-dns-latency-exporter-dkvxs -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
[...]
openshift.io/scc: restricted-v2
Additional info:
It's not yet clear to me how this can be solved respectively or what is actually triggering that issue. But we need to address that as otherwise it will expose problems when updating to OpenShift Container Platform 4.11 in examples like documetned above with AMQ Streams.