Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4224

ES pod throws "Not yet initialized" error

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • NEW
    • Bug Fix
    • Log Storage - Sprint 238

      Description of the problem: When CL instance is deployed with vector/fluentd, ES pod throws "Not yet initialized" error.

      Image used for testing: registry.redhat.io/openshift-logging/elasticsearch6-rhel9@sha256:0d177fe963c64df95d85af428e1e58afae06c7fbafe1fe406d7b9bb073024511

      Version: Logging 5.8.0

      How reproducible: Always 

      Steps to Reproduce:

      1. Install logging 5.8.0 CLO and EO
      2. Create CL instance with below spec:

      spec:
        managementState: "Managed"  
        logStore:
          type: "elasticsearch"  
          retentionPolicy: 
            application:
              maxAge: 10h
            infra:
              maxAge: 10h
            audit:
              maxAge: 10h
          elasticsearch:
            nodeCount: 1 
            storage: {} 
            resources: 
                limits:
                  memory: "4Gi"
                requests:
                  memory: "1Gi"
            proxy: 
              resources:
                limits:
                  memory: 256Mi
                requests:
                  memory: 256Mi
            redundancyPolicy: "ZeroRedundancy"
        visualization:
          type: "kibana"  
          kibana:
            replicas: 1
        collection:
          type: vector

      3. 

      cluster-logging-operator-fb96bbc6f-52blf        1/1     Running   0          78m
      collector-2c97t                                 1/1     Running   0          38m
      collector-8dd6j                                 1/1     Running   0          38m
      collector-d5g9r                                 1/1     Running   0          38m
      collector-n5zvj                                 1/1     Running   0          38m
      collector-nfd8z                                 1/1     Running   0          38m
      collector-vd2h7                                 1/1     Running   0          38m
      elasticsearch-cdm-d0vmsczo-1-7bf948b7d9-9tvsq   1/2     Running   0          18m
      elasticsearch-im-app-28109265-h5rpx             0/1     Error     0          8m21s
      elasticsearch-im-audit-28109265-x7cwj           0/1     Error     0          8m21s
      elasticsearch-im-infra-28109265-wgwkt           0/1     Error     0          8m21s

      Actual results:

      1. Seeing below error logs in ES pods:

      ....
      [2023-06-12 07:05:07,521][INFO ][container.run            ] Elasticsearch is ready and listening
      /usr/share/elasticsearch/init ~
      [2023-06-12 07:05:07,532][INFO ][container.run            ] Starting init script: 0001-jaeger
      [2023-06-12 07:05:07,536][INFO ][container.run            ] Completed init script: 0001-jaeger
      /usr/local/bin/es_util_env: line 19: python: command not found
      [2023-06-12 07:05:07,542][INFO ][container.run            ] Forcing the seeding of ACL documents
      [2023-06-12 07:05:07,701][INFO ][container.run            ] Seeding the security ACL index.  Will wait up to 604800 seconds.
      [2023-06-12 07:05:07,703][INFO ][container.run            ] Seeding the security ACL index.  Will wait up to 604800 seconds.
      /usr/local/bin/es_util_env: line 19: python: command not found
      Picked up _JAVA_OPTIONS: -Dcom.redhat.fips=false
      Open Distro Security Admin v6
      ERR: Parsing failed.  Reason: Missing argument for option: i
      usage: securityadmin.sh [-arc] [-cacert <file>] [-cd <directory>] [-cert
      ...
      ~
      [2023-06-12T07:05:23,380][ERROR][c.a.o.s.a.BackendRegistry] [elasticsearch-cdm-d0vmsczo-1] Not yet initialized 
      [2023-06-12T07:05:28,487][ERROR][c.a.o.s.a.BackendRegistry] [elasticsearch-cdm-d0vmsczo-1] Not yet initialized
      
      $ oc get deploy
      NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
      cluster-logging-operator       1/1     1            1           101m
      elasticsearch-cdm-d0vmsczo-1   0/1     1            0           71m
      kibana                         1/1     1            1           71m

            rh-ee-calee Calvin Lee
            gkarager Giriyamma Karagere Ramaswamy (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: