Undefined Description of problem:
When a Kafka instance is created on OpenShift version 4.16, we see a perpetual creation of dockercfg secrets in the namespace. In few hours the number can go upto 3000. Example of the secrets been generated.
my-cluster-entity-operator-dockercfg-2p8p9 kubernetes.io/dockercfg 1 11s my-cluster-entity-operator-dockercfg-f65c5 kubernetes.io/dockercfg 1 2m11s my-cluster-entity-operator-dockercfg-hp2kq kubernetes.io/dockercfg 1 10m my-cluster-entity-operator-dockercfg-mpg9f kubernetes.io/dockercfg 1 8m11s my-cluster-entity-operator-dockercfg-nxn6z kubernetes.io/dockercfg 1 6m11s my-cluster-entity-operator-dockercfg-pwtxd kubernetes.io/dockercfg 1 11m my-cluster-entity-operator-dockercfg-qk8bk kubernetes.io/dockercfg 1 10m my-cluster-entity-operator-dockercfg-t5ml5 kubernetes.io/dockercfg 1 4m11s my-cluster-entity-topic-operator-certs Opaque 4 11m my-cluster-entity-user-operator-certs Opaque 4 11m my-cluster-kafka-brokers Opaque 4 11m my-cluster-kafka-dockercfg-6hbmh kubernetes.io/dockercfg 1 2m12s my-cluster-kafka-dockercfg-7662g kubernetes.io/dockercfg 1 6m12s my-cluster-kafka-dockercfg-7lcmr kubernetes.io/dockercfg 1 8m12s my-cluster-kafka-dockercfg-7zvfd kubernetes.io/dockercfg 1 10m my-cluster-kafka-dockercfg-m9pfz kubernetes.io/dockercfg 1 4m12s my-cluster-kafka-dockercfg-nv2xk kubernetes.io/dockercfg 1 12s my-cluster-kafka-dockercfg-rp5dp kubernetes.io/dockercfg 1 11m my-cluster-kafka-dockercfg-xmns2 kubernetes.io/dockercfg 1 10m my-cluster-zookeeper-dockercfg-5f9lw kubernetes.io/dockercfg 1 13s my-cluster-zookeeper-dockercfg-7kt9s kubernetes.io/dockercfg 1 2m13s my-cluster-zookeeper-dockercfg-84pg8 kubernetes.io/dockercfg 1 6m13s my-cluster-zookeeper-dockercfg-cmr98 kubernetes.io/dockercfg 1 12m my-cluster-zookeeper-dockercfg-f2xz4 kubernetes.io/dockercfg 1 4m13s my-cluster-zookeeper-dockercfg-hs68r kubernetes.io/dockercfg 1 8m13s my-cluster-zookeeper-dockercfg-v8q4s kubernetes.io/dockercfg 1 10m my-cluster-zookeeper-dockercfg-w6trk kubernetes.io/dockercfg 1 10m
Version-Release number of selected component (if applicable):
amqstreams.v2.7.0-2
Steps to Reproduce:
*Create a Kafka CR instance and create some topics.
apiVersion: v1 kind: Namespace metadata: name: chainsaw-kafka --- apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster namespace: chainsaw-kafka spec: entityOperator: topicOperator: reconciliationIntervalSeconds: 90 userOperator: reconciliationIntervalSeconds: 120 kafka: config: log.message.format.version: 3.7.0 message.max.bytes: 10485760 offsets.topic.replication.factor: 1 ssl.cipher.suites: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ssl.enabled.protocols: TLSv1.2 ssl.protocol: TLSv1.2 transaction.state.log.min.isr: 1 transaction.state.log.replication.factor: 1 jvmOptions: -Xms: 1024m -Xmx: 1024m listeners: - configuration: useServiceDnsDomain: true name: plain port: 9092 tls: false type: internal - authentication: type: tls name: tls port: 9093 tls: true type: internal replicas: 1 resources: limits: cpu: "1" memory: 4Gi requests: cpu: "1" memory: 4Gi storage: type: ephemeral version: 3.7.0 zookeeper: replicas: 1 storage: type: ephemeral apiVersion: kafka.strimzi.io/v1beta1 kind: KafkaTopic metadata: labels: strimzi.io/cluster: my-cluster name: otlp-spans namespace: chainsaw-kafka spec: config: retention.ms: 300000 segment.bytes: 1073741824 partitions: 1 replicas: 1
*Let the instance run. Observe that the Kafka, Zookeeper and Entity operator secrets having dockercfg starts increasing.
% oc get secrets NAME TYPE DATA AGE builder-dockercfg-q89fv kubernetes.io/dockercfg 1 12m default-dockercfg-ks242 kubernetes.io/dockercfg 1 12m deployer-dockercfg-ftlcp kubernetes.io/dockercfg 1 12m my-cluster-clients-ca Opaque 1 12m my-cluster-clients-ca-cert Opaque 3 12m my-cluster-cluster-ca Opaque 1 12m my-cluster-cluster-ca-cert Opaque 3 12m my-cluster-cluster-operator-certs Opaque 4 12m my-cluster-entity-operator-dockercfg-2p8p9 kubernetes.io/dockercfg 1 11s my-cluster-entity-operator-dockercfg-f65c5 kubernetes.io/dockercfg 1 2m11s my-cluster-entity-operator-dockercfg-hp2kq kubernetes.io/dockercfg 1 10m my-cluster-entity-operator-dockercfg-mpg9f kubernetes.io/dockercfg 1 8m11s my-cluster-entity-operator-dockercfg-nxn6z kubernetes.io/dockercfg 1 6m11s my-cluster-entity-operator-dockercfg-pwtxd kubernetes.io/dockercfg 1 11m my-cluster-entity-operator-dockercfg-qk8bk kubernetes.io/dockercfg 1 10m my-cluster-entity-operator-dockercfg-t5ml5 kubernetes.io/dockercfg 1 4m11s my-cluster-entity-topic-operator-certs Opaque 4 11m my-cluster-entity-user-operator-certs Opaque 4 11m my-cluster-kafka-brokers Opaque 4 11m my-cluster-kafka-dockercfg-6hbmh kubernetes.io/dockercfg 1 2m12s my-cluster-kafka-dockercfg-7662g kubernetes.io/dockercfg 1 6m12s my-cluster-kafka-dockercfg-7lcmr kubernetes.io/dockercfg 1 8m12s my-cluster-kafka-dockercfg-7zvfd kubernetes.io/dockercfg 1 10m my-cluster-kafka-dockercfg-m9pfz kubernetes.io/dockercfg 1 4m12s my-cluster-kafka-dockercfg-nv2xk kubernetes.io/dockercfg 1 12s my-cluster-kafka-dockercfg-rp5dp kubernetes.io/dockercfg 1 11m my-cluster-kafka-dockercfg-xmns2 kubernetes.io/dockercfg 1 10m my-cluster-zookeeper-dockercfg-5f9lw kubernetes.io/dockercfg 1 13s my-cluster-zookeeper-dockercfg-7kt9s kubernetes.io/dockercfg 1 2m13s my-cluster-zookeeper-dockercfg-84pg8 kubernetes.io/dockercfg 1 6m13s my-cluster-zookeeper-dockercfg-cmr98 kubernetes.io/dockercfg 1 12m my-cluster-zookeeper-dockercfg-f2xz4 kubernetes.io/dockercfg 1 4m13s my-cluster-zookeeper-dockercfg-hs68r kubernetes.io/dockercfg 1 8m13s my-cluster-zookeeper-dockercfg-v8q4s kubernetes.io/dockercfg 1 10m my-cluster-zookeeper-dockercfg-w6trk kubernetes.io/dockercfg 1 10m my-cluster-zookeeper-nodes Opaque 4 12m
*The generated secrets can increase upto thousands over days.
Additional info:
The annotation openshift.io/internal-registry-pull-secret-ref is now added to the SA in OCP 4.16 which the operator tries to reconcile causing a loop.
- relates to
-
OCPBUGS-36833 4.16 "Bad" reconciliation loops can cause unbounded dockercfg secret creation
-
- Closed
-
- links to
-
RHSA-2024:142550
Streams for Apache Kafka 2.8.0 release and security update
