-
Bug
-
Resolution: Done
-
Normal
-
None
-
3
-
False
-
-
False
-
?
-
?
-
?
-
?
-
None
-
-
-
DFG Security: UC Sprint 98, DFG Security: UC Sprint 99, DFG Security: UC Sprint 100, DFG Security: UC Sprint 101
-
Important
This is clone of OSPRH-7992, Consider using VerifySecret as mentioned in this comment
Adding a new field to the osp-secret triggers mass pod restarts on the control plane.
$ oc set data secret/osp-secret foo=bar
$ oc get pods -n openstack --watch ... glance-default-single-0 3/3 Terminating 0 15h cinder-scheduler-0 2/2 Terminating 0 15h barbican-worker-worker-66cc7748cb-lhbs2 0/2 Pending 0 0s keystone-69c6b545b5-l72jc 0/1 Pending 0 0s barbican-keystone-listener-keystone-listener-7c8f4f484b-hw2mc 0/2 Pending 0 0s barbican-api-api-f4b8c966-2s2pd 0/2 Pending 0 0s barbican-worker-worker-66cc7748cb-lhbs2 0/2 Pending 0 0s keystone-69c6b545b5-l72jc 0/1 Pending 0 0s barbican-api-api-f4b8c966-2s2pd 0/2 Pending 0 0s barbican-keystone-listener-keystone-listener-7c8f4f484b-hw2mc 0/2 Pending 0 0s barbican-api-api-f4b8c966-2s2pd 0/2 Pending 0 0s barbican-worker-worker-66cc7748cb-lhbs2 0/2 ContainerCreating 0 0s barbican-keystone-listener-keystone-listener-7c8f4f484b-hw2mc 0/2 Pending 0 0s barbican-keystone-listener-keystone-listener-7c8f4f484b-hw2mc 0/2 ContainerCreating 0 0s cinder-api-0 2/2 Terminating 1 (40m ago) 15h barbican-worker-worker-66cc7748cb-lhbs2 0/2 ContainerCreating 0 0s keystone-69c6b545b5-l72jc 0/1 Pending 0 0s neutron-864d7467ff-mcn5t 0/2 Pending 0 0s barbican-api-api-f4b8c966-2s2pd 0/2 ContainerCreating 0 0s neutron-864d7467ff-mcn5t 0/2 Pending 0 0s neutron-864d7467ff-mcn5t 0/2 Pending 0 0s keystone-69c6b545b5-l72jc 0/1 ContainerCreating 0 0s neutron-864d7467ff-mcn5t 0/2 ContainerCreating 0 0s barbican-api-api-f4b8c966-2s2pd 0/2 ContainerCreating 0 0s barbican-keystone-listener-keystone-listener-7c8f4f484b-hw2mc 0/2 ContainerCreating 0 1s keystone-69c6b545b5-l72jc 0/1 ContainerCreating 0 1s barbican-worker-worker-66cc7748cb-lhbs2 0/2 ContainerCreating 0 1s neutron-864d7467ff-mcn5t 0/2 ContainerCreating 0 1s barbican-keystone-listener-keystone-listener-7c8f4f484b-hw2mc 2/2 Running 0 1s barbican-api-api-f4b8c966-2s2pd 0/2 Running 0 1s barbican-keystone-listener-keystone-listener-d77c5446d-jlcjk 2/2 Terminating 0 15h barbican-worker-worker-66cc7748cb-lhbs2 2/2 Running 0 1s keystone-69c6b545b5-l72jc 0/1 Running 0 1s barbican-worker-worker-7c6f766576-9z4hg 2/2 Terminating 0 15h swift-proxy-6f844fddb-ss7d9 0/2 Pending 0 0s swift-proxy-6f844fddb-ss7d9 0/2 Pending 0 0s swift-proxy-6f844fddb-ss7d9 0/2 Pending 0 0s swift-proxy-6f844fddb-ss7d9 0/2 ContainerCreating 0 0s neutron-864d7467ff-mcn5t 0/2 Running 0 2s glance-default-single-0 0/3 Terminating 0 15h swift-proxy-6f844fddb-ss7d9 0/2 ContainerCreating 0 1s cinder-scheduler-0 0/2 Terminating 0 15h cinder-scheduler-0 0/2 Terminating 0 15h swift-proxy-6f844fddb-ss7d9 0/2 Running 0 1s glance-default-single-0 0/3 Terminating 0 15h glance-default-single-0 0/3 Terminating 0 15h glance-default-single-0 0/3 Terminating 0 15h cinder-scheduler-0 0/2 Terminating 0 15h cinder-scheduler-0 0/2 Terminating 0 15h glance-default-single-0 0/3 Pending 0 1s glance-default-single-0 0/3 Pending 0 1s cinder-scheduler-0 0/2 Pending 0 0s cinder-scheduler-0 0/2 Pending 0 0s glance-default-single-0 0/3 Pending 0 1s cinder-scheduler-0 0/2 Pending 0 0s glance-default-single-0 0/3 ContainerCreating 0 1s cinder-scheduler-0 0/2 ContainerCreating 0 0s cinder-api-0 0/2 Terminating 1 (41m ago) 15h cinder-scheduler-0 0/2 ContainerCreating 0 0s glance-default-single-0 0/3 ContainerCreating 0 1s cinder-api-0 0/2 Terminating 1 (41m ago) 15h cinder-api-0 0/2 Terminating 1 (41m ago) 15h cinder-api-0 0/2 Terminating 1 (41m ago) 15h cinder-api-0 0/2 Pending 0 0s cinder-api-0 0/2 Pending 0 0s cinder-api-0 0/2 Pending 0 0s cinder-api-0 0/2 ContainerCreating 0 0s cinder-api-0 0/2 ContainerCreating 0 0s cinder-scheduler-0 1/2 Running 0 1s glance-default-single-0 0/3 Running 0 2s cinder-api-0 1/2 Running 0 1s barbican-worker-worker-7c6f766576-9z4hg 0/2 Terminating 0 15h ceilometer-0 4/4 Terminating 0 15h barbican-keystone-listener-keystone-listener-d77c5446d-jlcjk 0/2 Terminating 0 15h barbican-worker-worker-7c6f766576-9z4hg 0/2 Terminating 0 15h barbican-worker-worker-7c6f766576-9z4hg 0/2 Terminating 0 15h barbican-worker-worker-7c6f766576-9z4hg 0/2 Terminating 0 15h barbican-keystone-listener-keystone-listener-d77c5446d-jlcjk 0/2 Terminating 0 15h barbican-keystone-listener-keystone-listener-d77c5446d-jlcjk 0/2 Terminating 0 15h barbican-keystone-listener-keystone-listener-d77c5446d-jlcjk 0/2 Terminating 0 15h swift-proxy-6f844fddb-ss7d9 2/2 Running 0 6s swift-proxy-6bf6c878f8-rxvt9 2/2 Terminating 0 15h swift-proxy-6bf6c878f8-rxvt9 0/2 Terminating 0 15h swift-proxy-6bf6c878f8-rxvt9 0/2 Terminating 0 15h swift-proxy-6bf6c878f8-rxvt9 0/2 Terminating 0 15h swift-proxy-6bf6c878f8-rxvt9 0/2 Terminating 0 15h barbican-api-api-f4b8c966-2s2pd 1/2 Running 0 11s barbican-api-api-f4b8c966-2s2pd 2/2 Running 0 11s barbican-api-api-b466bbb8-m894j 2/2 Terminating 0 15h glance-default-single-0 0/3 Running 0 11s glance-default-single-0 0/3 Running 0 11s cinder-scheduler-0 1/2 Running 0 10s cinder-scheduler-0 2/2 Running 0 10s barbican-api-api-b466bbb8-m894j 0/2 Terminating 0 15h ceilometer-0 0/4 Terminating 0 15h barbican-api-api-b466bbb8-m894j 0/2 Terminating 0 15h ceilometer-0 0/4 Terminating 0 15h ceilometer-0 0/4 Terminating 0 15h ceilometer-0 0/4 Terminating 0 15h barbican-api-api-b466bbb8-m894j 0/2 Terminating 0 15h barbican-api-api-b466bbb8-m894j 0/2 Terminating 0 15h ceilometer-0 0/4 Pending 0 0s ceilometer-0 0/4 Pending 0 0s ceilometer-0 0/4 Pending 0 0s ceilometer-0 0/4 ContainerCreating 0 0s glance-default-single-0 1/3 Running 0 12s glance-default-single-0 2/3 Running 0 12s glance-default-single-0 3/3 Running 0 12s ceilometer-0 0/4 ContainerCreating 0 0s cinder-api-0 2/2 Running 0 11s ceilometer-0 3/4 Running 0 7s neutron-864d7467ff-mcn5t 1/2 Running 0 25s neutron-864d7467ff-mcn5t 2/2 Running 0 25s neutron-6dcb97bd77-ppzsn 2/2 Terminating 0 15h keystone-69c6b545b5-l72jc 1/1 Running 0 31s keystone-56fc456c54-jpz25 1/1 Terminating 0 15h keystone-56fc456c54-jpz25 0/1 Terminating 0 15h keystone-56fc456c54-jpz25 0/1 Terminating 0 15h keystone-56fc456c54-jpz25 0/1 Terminating 0 15h keystone-56fc456c54-jpz25 0/1 Terminating 0 15h neutron-6dcb97bd77-ppzsn 1/2 Terminating 0 15h neutron-6dcb97bd77-ppzsn 0/2 Terminating 0 15h ceilometer-0 4/4 Running 0 30s neutron-6dcb97bd77-ppzsn 0/2 Terminating 0 15h neutron-6dcb97bd77-ppzsn 0/2 Terminating 0 15h neutron-6dcb97bd77-ppzsn 0/2 Terminating 0 15h neutron-6dcb97bd77-ppzsn 0/2 Terminating 0 15h
It seems that most of the service operators restart the service pods even if an unrelated field is add/modified in the osp-secret causing an unnecessary control plane outage. I think this is caused by the logic in these service operators to create the deployments to depend on the hash of the whole secret instead of only depending on the hash of the key-value pairs from the secret the deployed service actually uses. See the difference between nova-operator (not affected) and keyston-operator (affected):
- https://github.com/openstack-k8s-operators/nova-operator/blob/main/controllers/common.go#L188-L190
- https://github.com/openstack-k8s-operators/keystone-operator/blob/04190feb45970bbe96ac1b7961eb7b2904cb5903/controllers/keystoneapi_controller.go#L713-L731
Workaround: do not use a single centralized osp-secret file in our documentation but instead pass a separate secret to each service via the OpenStackControlPlane CR.
- clones
-
-
- Closed
-
