Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-3428

v5.5.6-8 cluster-logging-operator pod ends up in CrashLoopBackOff status

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide
      Before this change a fix was added to support a regression where the status of the elasticsearch deployment was not being updated. This caused the operator to crash unless the Red Hat elasticsearch-operator was deployed. This change reverts that fix so the operator is now stable but re-introduces the previous issue related to the reported status.
      Show
      Before this change a fix was added to support a regression where the status of the elasticsearch deployment was not being updated. This caused the operator to crash unless the Red Hat elasticsearch-operator was deployed. This change reverts that fix so the operator is now stable but re-introduces the previous issue related to the reported status.
    • Rox Sprint 74C - Global, Log Collection - Sprint 229
    • Important

    Description

      Description of problem:

      v5.5.6-8 cluster-logging-operator pod ends up in CrashLoopBackOff status.

      [kni@registry.kni-qe-0 ~]$ oc -n openshift-logging get pods

       

      NAME                                      READY   STATUS             RESTARTS         AGE
      cluster-logging-operator-7cbd77bf-r5ksv   0/1     CrashLoopBackOff   12 (3m28s ago)   69m
      collector-tnmqg                           2/2     Running            0                68m
      

       

      [kni@registry.kni-qe-0 ~]$ oc -n openshift-logging logs cluster-logging-operator-7cbd77bf-r5ksv

       

      {"_ts":"2022-12-14T11:11:37.225541809Z","_level":"0","_component":"cluster-logging-operator","_message":"starting up...","go_arch":"amd64","go_os":"linux","go_version":"go1.17.12","operator_version":"5.5"}
      I1214 11:11:38.276655       1 request.go:665] Waited for 1.039760579s due to client-side throttling, not priority and fairness, request: GET:https://[fd02::1]:443/apis/security.internal.openshift.io/v1?timeout=32s
      {"_ts":"2022-12-14T11:11:39.932221196Z","_level":"0","_component":"cluster-logging-operator","_message":"migrating resources provided by the manifest"} {"_ts":"2022-12-14T11:11:39.934778181Z","_level":"0","_component":"cluster-logging-operator","_message":"Registering Components."} {"_ts":"2022-12-14T11:11:39.935175808Z","_level":"0","_component":"cluster-logging-operator","_message":"Starting the Cmd."}
      I1214 11:11:55.186275       1 request.go:665] Waited for 1.046884604s due to client-side throttling, not priority and fairness, request: GET:https://[fd02::1]:443/apis/image.openshift.io/v1?timeout=32s
      I1214 11:12:05.187029       1 request.go:665] Waited for 1.045809582s due to client-side throttling, not priority and fairness, request: GET:https://[fd02::1]:443/apis/machine.openshift.io/v1beta1?timeout=32s
      I1214 11:12:15.236201       1 request.go:665] Waited for 1.097533537s due to client-side throttling, not priority and fairness, request: GET:https://[fd02::1]:443/apis/authorization.k8s.io/v1?timeout=32s
      I1214 11:12:25.236923       1 request.go:665] Waited for 1.096393882s due to client-side throttling, not priority and fairness, request: GET:https://[fd02::1]:443/apis/project.openshift.io/v1?timeout=32s
      I1214 11:12:35.286178       1 request.go:665] Waited for 1.146796421s due to client-side throttling, not priority and fairness, request: GET:https://[fd02::1]:443/apis/autoscaling/v2beta2?timeout=32s
      I1214 11:12:45.336396       1 request.go:665] Waited for 1.197389464s due to client-side throttling, not priority and fairness, request: GET:https://[fd02::1]:443/apis/local.storage.openshift.io/v1?timeout=32s
      I1214 11:12:55.336482       1 request.go:665] Waited for 1.198072466s due to client-side throttling, not priority and fairness, request: GET:https://[fd02::1]:443/apis/coordination.k8s.io/v1?timeout=32s
      I1214 11:13:05.387352       1 request.go:665] Waited for 1.224493028s due to client-side throttling, not priority and fairness, request: GET:https://[fd02::1]:443/apis/local.storage.openshift.io/v1?timeout=32s
      {"_ts":"2022-12-14T11:13:11.017893653Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogforwarder-controller error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterlogforwarders.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}}
      I1214 11:13:15.437612       1 request.go:665] Waited for 1.22207017s due to client-side throttling, not priority and fairness, request: GET:https://[fd02::1]:443/apis/operators.coreos.com/v1?timeout=32s
      I1214 11:13:25.486911       1 request.go:665] Waited for 1.348064753s due to client-side throttling, not priority and fairness, request: GET:https://[fd02::1]:443/apis/sriovnetwork.openshift.io/v1?timeout=32s
      I1214 11:13:35.536707       1 request.go:665] Waited for 1.397478125s due to client-side throttling, not priority and fairness, request: GET:https://[fd02::1]:443/apis/performance.openshift.io/v1alpha1?timeout=32s
      {"_ts":"2022-12-14T11:13:36.889681986Z","_level":"0","_component":"cluster-logging-operator","_message":"clusterlogforwarder-controller error updating status","_error":{"msg":"Operation cannot be fulfilled on clusterlogforwarders.logging.openshift.io \"instance\": the object has been modified; please apply your changes to the latest version and try again"}}
      I1214 11:13:45.536951       1 request.go:665] Waited for 1.397949876s due to client-side throttling, not priority and fairness, request: GET:https://[fd02::1]:443/apis/admissionregistration.k8s.io/v1?timeout=32s
      {"_ts":"2022-12-14T11:14:01.024031054Z","_level":"0","_component":"cluster-logging-operator","_message":"Manager exited non-zero","_error":{"msg":"failed to wait for clusterlogging caches to sync: timed out waiting for cache to be synced"}}
      

       

      Version-Release number of selected component (if applicable):

      cluster-logging-operator-metadata-container-v5.5.6-8

      registry-proxy.engineering.redhat.com/rh-osbs/openshift-logging-cluster-logging-rhel8-operator:v5.5.6-4

      {
         .......
        "architecture": "x86_64",
        "build-date": "2022-12-13T15:25:42",
        "io.openshift.build.commit.id": "ce22bf231b89df7923dc45666cb220a8c29eca1b",
        "io.openshift.build.commit.url": "https://github.com/openshift/cluster-logging-operator/commit/ce22bf231b89df7923dc45666cb220a8c29eca1b",
        "name": "openshift-logging/cluster-logging-rhel8-operator",
        "release": "4",
        "summary": "Provides the latest release of Red Hat Universal Base Image 8.",
        "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/openshift-logging/cluster-logging-rhel8-operator/images/v5.5.6-4",
        "vcs-ref": "f1d8d61ee49159769e555234548ce7a93d24fa46",
        "vcs-type": "git",
        "vendor": "Red Hat, Inc.",
        "version": "v5.5.6"
      }

      4.12.0-rc.4,  Single node and IPv6 Single stack cluster

       

      How reproducible:

      always

      Steps to Reproduce:

      1.   Install and configure cluster-logging on an SNO IPv6 environment with Telco DU profile applied
      2.  Check cluster-logging-operator  pod status

      Actual results:

       

      Pod gets restarted and ends up in CrashLoopBackOff

      Expected results:

       

      No failures.

      Additional info:

       

      Attaching output of operator image info and pod describe.pod-info.txt

       

       

       

       

      Attachments

        Issue Links

          Activity

            People

              jcantril@redhat.com Jeffrey Cantrill
              mcornea@redhat.com Marius Cornea
              Anping Li Anping Li
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: