Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-5007

ClusterLogForwarder's status is constantly replaced, leading to infinite reconciliation runs

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • NEW
    • Hide
      In specific corner cases, replacing the CLF status field caused the resourceVersion to be updated constantly due to changing timestamps in Status conditions. This in turn lead to an infinite reconciliation loop. Instead, synchronize all status conditions so that timestamps remain unchanged if conditions are not changed.
      Show
      In specific corner cases, replacing the CLF status field caused the resourceVersion to be updated constantly due to changing timestamps in Status conditions. This in turn lead to an infinite reconciliation loop. Instead, synchronize all status conditions so that timestamps remain unchanged if conditions are not changed.
    • Bug Fix
    • Log Collection - Sprint 249, Log Collection - Sprint 250
    • Low

      This affects the old / legacy implementation of the forwarder. For the upstream bug, see: https://github.com/openshift/cluster-logging-operator/issues/2315

      ClusterLogForwarder's status is constantly replaced, leading to infinite reconciliation runs

      The following code:

      controllers/forwarding/forwarding_controller.go
@@ -83,7 +83,9 @@ func (r *ReconcileForwarder) Reconcile(ctx context.Context, request ctrl.Request
        // Fetch the ClusterLogForwarder instance
        instance, err, status := loader.FetchClusterLogForwarder(r.Client, request.NamespacedName.Namespace, request.NamespacedName.Name, true, func() logging.ClusterLogging { return *cl })
        if status != nil {
              instance.Status = *status

      always repalces the entire CLF status. That in turn leads to constant status updates of the type Ready condition where lastTransitionTime is always set, even though no actual transition happens.

      In turn, the operator infinitely reconciles resources, even though nothing happens.

      You can easily see this by running the current version of the Operator with debug level 3:

       LOG_LEVEL=3 make run

      With the following 2 resources in place:

      [akaris@workstation logging]$ cat configuring-cluster-logging-minimal.yaml
apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance" 
  namespace: openshift-logging
spec:
  managementState: "Managed"
  collection:
    tolerations:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
      operator: Exists
    type: vector
[akaris@workstation logging]$ cat configuring-log-forwarder.yaml
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  pipelines:
   - inputRefs:
     - audit
     outputRefs:
     - loki-external
     labels:
       job: openshift-audit
       clusterid: sno10
   - inputRefs:
     - application
     outputRefs:
     - loki-external
     labels:
       job: openshift-application
       clusterid: sno10
   - inputRefs:
     - infrastructure
     outputRefs:
     - loki-external
     labels:
       job: openshift-infrastructure
       clusterid: sno10
  outputs:
  - name: loki-external
    type: loki

              akaris@redhat.com Andreas Karis
              akaris@redhat.com Andreas Karis
              Anping Li Anping Li
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: