Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-5776

Continuously generating secrets in the Elasticsearch, Kibana instance namespace on OCP 4.16

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Critical Critical
    • Logging 5.8.9
    • None
    • Log Storage
    • None
    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide
      Before this update, the elasticsearch-operator overwrote all ServiceAccount annotations without considering ownership caused the kube-controler-manager to recreate serviceaccount secrets because the link to the owning serviceaccount got logs. With this update, the elasticsearch-operator merging annotations resolves the issue.
      Show
      Before this update, the elasticsearch-operator overwrote all ServiceAccount annotations without considering ownership caused the kube-controler-manager to recreate serviceaccount secrets because the link to the owning serviceaccount got logs. With this update, the elasticsearch-operator merging annotations resolves the issue.
    • Bug Fix
    • Log Storage - Sprint 256

      Description of problem:

      When a Elasticsearch instance is created on OpenShift version 4.16, we see a perpetual creation of dockercfg secrets in the namespace. In few hours the number can go upto 3000. Example of the secrets been generated. 

      elasticsearch-dockercfg-42j49                 kubernetes.io/dockercfg               1      108s
      elasticsearch-dockercfg-64czc                 kubernetes.io/dockercfg               1      2m18s
      elasticsearch-dockercfg-7jlmv                 kubernetes.io/dockercfg               1      108s
      elasticsearch-dockercfg-86rr2                 kubernetes.io/dockercfg               1      48s
      elasticsearch-dockercfg-g4kk8                 kubernetes.io/dockercfg               1      78s
      elasticsearch-dockercfg-pf8n2                 kubernetes.io/dockercfg               1      16s
      elasticsearch-dockercfg-q7rzc                 kubernetes.io/dockercfg               1      47s
      elasticsearch-dockercfg-tq6xk                 kubernetes.io/dockercfg               1      17s
      elasticsearch-dockercfg-xz8j2                 kubernetes.io/dockercfg               1      2m20s 

      Version-Release number of selected component (if applicable):

      cluster-logging.v5.9.3

      elasticsearch-operator.v5.8.3 

      OCP Server Version: 4.16.1

      How reproducible:

      Always

      Steps to Reproduce:

      1. Create a Cluster Logging instance with Elasticsearch as log store. Elasticsearch operator is a dependency of other operators like Jaeger and if we create a Jaeger instance with mode production or streaming using Elasticsearch, the issue can be reproduced as well. 

      % oc get clusterversion 
      NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.16.1    True        False         19m     Cluster version is 4.16.1
      
      % oc get csv
      NAME                                DISPLAY                                          VERSION     REPLACES                            PHASE
      amqstreams.v2.7.0-2                 Red Hat Streams for Apache Kafka                 2.7.0-2     amqstreams.v2.7.0-1                 Succeeded
      cluster-logging.v5.9.3              Red Hat OpenShift Logging                        5.9.3       cluster-logging.v5.9.2              Succeeded
      elasticsearch-operator.v5.8.3       OpenShift Elasticsearch Operator                 5.8.3       elasticsearch-operator.v5.8.2       Succeeded
      jaeger-operator.v1.57.0-6           Red Hat OpenShift distributed tracing platform   1.57.0-6    jaeger-operator.v1.57.0-5           Succeeded
      kiali-operator.v1.73.8              Kiali Operator                                   1.73.8      kiali-operator.v1.73.7              Succeeded
      loki-operator.v5.9.3                Loki Operator                                    5.9.3       loki-operator.v5.9.2                Succeeded
      opentelemetry-operator.v0.102.0-2   Red Hat build of OpenTelemetry                   0.102.0-2   opentelemetry-operator.v0.100.1-3   Succeeded
      servicemeshoperator.v2.5.2          Red Hat OpenShift Service Mesh                   2.5.2-0     servicemeshoperator.v2.5.1          Succeeded
      tempo-operator.v0.10.0-7            Tempo Operator                                   0.10.0-7    tempo-operator.v0.10.0-6            Succeeded
      
      % cat install-logging-with-es.yaml 
      apiVersion: logging.openshift.io/v1
      kind: ClusterLogging
      metadata:
        name: instance 
        namespace: openshift-logging
      spec:
        managementState: Managed 
        logStore:
          type: elasticsearch 
          retentionPolicy: 
            application:
              maxAge: 1d
            infra:
              maxAge: 7d
            audit:
              maxAge: 7d
          elasticsearch:
            nodeCount: 3 
            storage:
              storageClassName: gp2-csi
              size: 20G
            resources: 
                requests:
                  memory: 4Gi
            proxy: 
              resources:
                requests:
                  memory: 256Mi
            redundancyPolicy: SingleRedundancy
        visualization:
          type: kibana 
          kibana:
            replicas: 1
        collection:
          type: vector 
      
      % oc get pods
      NAME                                            READY   STATUS    RESTARTS   AGE
      cluster-logging-operator-8545947b7-4xv4b        1/1     Running   0          7m31s
      collector-4sk89                                 1/1     Running   0          5m12s
      collector-mjsnp                                 1/1     Running   0          5m12s
      collector-n667v                                 1/1     Running   0          5m12s
      collector-v2pbq                                 1/1     Running   0          5m12s
      collector-wpvpb                                 1/1     Running   0          5m12s
      collector-z8s7f                                 1/1     Running   0          5m12s
      elasticsearch-cdm-c70qkt62-1-698cbd8f7d-576lf   2/2     Running   0          5m6s
      elasticsearch-cdm-c70qkt62-2-76cb8859d9-rb428   2/2     Running   0          5m5s
      elasticsearch-cdm-c70qkt62-3-7b7fc88876-xrz85   2/2     Running   0          5m4s
      kibana-9dbd547dd-5l44g                          2/2     Running   0          4m56s

      *Check the secrets in the namespace, the Elasticsearch, Kibana secrets having dockercfg will start increasing overtime. 

      % oc get secrets| wc -l
           295 

      Additional info:

      The annotation openshift.io/internal-registry-pull-secret-ref is now added to the SA in OCP 4.16 which the operator tries to reconcile causing a loop.

            ptsiraki@redhat.com Periklis Tsirakidis
            rhn-support-ikanse Ishwar Kanse
            Qiaoling Tang Qiaoling Tang
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: