-
Bug
-
Resolution: Done-Errata
-
Critical
-
None
-
None
-
False
-
None
-
False
-
NEW
-
VERIFIED
-
-
Bug Fix
-
-
-
Log Storage - Sprint 256
Description of problem:
When a Elasticsearch instance is created on OpenShift version 4.16, we see a perpetual creation of dockercfg secrets in the namespace. In few hours the number can go upto 3000. Example of the secrets been generated.
elasticsearch-dockercfg-42j49 kubernetes.io/dockercfg 1 108s elasticsearch-dockercfg-64czc kubernetes.io/dockercfg 1 2m18s elasticsearch-dockercfg-7jlmv kubernetes.io/dockercfg 1 108s elasticsearch-dockercfg-86rr2 kubernetes.io/dockercfg 1 48s elasticsearch-dockercfg-g4kk8 kubernetes.io/dockercfg 1 78s elasticsearch-dockercfg-pf8n2 kubernetes.io/dockercfg 1 16s elasticsearch-dockercfg-q7rzc kubernetes.io/dockercfg 1 47s elasticsearch-dockercfg-tq6xk kubernetes.io/dockercfg 1 17s elasticsearch-dockercfg-xz8j2 kubernetes.io/dockercfg 1 2m20s
Version-Release number of selected component (if applicable):
cluster-logging.v5.9.3
elasticsearch-operator.v5.8.3
OCP Server Version: 4.16.1
How reproducible:
Always
Steps to Reproduce:
1. Create a Cluster Logging instance with Elasticsearch as log store. Elasticsearch operator is a dependency of other operators like Jaeger and if we create a Jaeger instance with mode production or streaming using Elasticsearch, the issue can be reproduced as well.
% oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.16.1 True False 19m Cluster version is 4.16.1 % oc get csv NAME DISPLAY VERSION REPLACES PHASE amqstreams.v2.7.0-2 Red Hat Streams for Apache Kafka 2.7.0-2 amqstreams.v2.7.0-1 Succeeded cluster-logging.v5.9.3 Red Hat OpenShift Logging 5.9.3 cluster-logging.v5.9.2 Succeeded elasticsearch-operator.v5.8.3 OpenShift Elasticsearch Operator 5.8.3 elasticsearch-operator.v5.8.2 Succeeded jaeger-operator.v1.57.0-6 Red Hat OpenShift distributed tracing platform 1.57.0-6 jaeger-operator.v1.57.0-5 Succeeded kiali-operator.v1.73.8 Kiali Operator 1.73.8 kiali-operator.v1.73.7 Succeeded loki-operator.v5.9.3 Loki Operator 5.9.3 loki-operator.v5.9.2 Succeeded opentelemetry-operator.v0.102.0-2 Red Hat build of OpenTelemetry 0.102.0-2 opentelemetry-operator.v0.100.1-3 Succeeded servicemeshoperator.v2.5.2 Red Hat OpenShift Service Mesh 2.5.2-0 servicemeshoperator.v2.5.1 Succeeded tempo-operator.v0.10.0-7 Tempo Operator 0.10.0-7 tempo-operator.v0.10.0-6 Succeeded % cat install-logging-with-es.yaml apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance namespace: openshift-logging spec: managementState: Managed logStore: type: elasticsearch retentionPolicy: application: maxAge: 1d infra: maxAge: 7d audit: maxAge: 7d elasticsearch: nodeCount: 3 storage: storageClassName: gp2-csi size: 20G resources: requests: memory: 4Gi proxy: resources: requests: memory: 256Mi redundancyPolicy: SingleRedundancy visualization: type: kibana kibana: replicas: 1 collection: type: vector % oc get pods NAME READY STATUS RESTARTS AGE cluster-logging-operator-8545947b7-4xv4b 1/1 Running 0 7m31s collector-4sk89 1/1 Running 0 5m12s collector-mjsnp 1/1 Running 0 5m12s collector-n667v 1/1 Running 0 5m12s collector-v2pbq 1/1 Running 0 5m12s collector-wpvpb 1/1 Running 0 5m12s collector-z8s7f 1/1 Running 0 5m12s elasticsearch-cdm-c70qkt62-1-698cbd8f7d-576lf 2/2 Running 0 5m6s elasticsearch-cdm-c70qkt62-2-76cb8859d9-rb428 2/2 Running 0 5m5s elasticsearch-cdm-c70qkt62-3-7b7fc88876-xrz85 2/2 Running 0 5m4s kibana-9dbd547dd-5l44g 2/2 Running 0 4m56s
*Check the secrets in the namespace, the Elasticsearch, Kibana secrets having dockercfg will start increasing overtime.
% oc get secrets| wc -l
295
Additional info:
The annotation openshift.io/internal-registry-pull-secret-ref is now added to the SA in OCP 4.16 which the operator tries to reconcile causing a loop.
- relates to
-
-
- Closed
-
- links to
-
RHBA-2024:4335
Logging for Red Hat OpenShift - 5.8.9
- mentioned on
