Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-6547

Collector pod takes a lot of time to start when syslog-receiver is enabled

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • NEW
    • Bug Fix
    • Moderate

      Description of problem:

      When a CLF is created with a syslog-receiver enabled, Collector pod is launched, but stays in "ContainerCreating" for a random amount of minutes. Currently we have measured between 5 and 45 minutes.

      If a CLF is created that does not contain a syslog-receiver, Collector pod starts in a couple of seconds.

      Version-Release number of selected component (if applicable):

      stable-6.1

      How reproducible:

      Always, with the OperatorHub installed stable-6.1 CLO

      Steps to Reproduce:

      1. Deploy and configure a 6.1.0 Logging environment
      2. Run CLF:
        1. apiVersion: observability.openshift.io/v1
          kind: ClusterLogForwarder
          metadata:
            name: collector
            namespace: openshift-logging
          spec:
            serviceAccount:
              name: collector
            inputs:
              - name: syslog-receiver
                type: receiver
                receiver:
                  type: syslog
                  port: 10514
            outputs:
              - name: default-lokistack
                type: lokiStack
                lokiStack:
                  authentication:
                    token:
                      from: serviceAccount
                  target:
                    name: logging-loki
                    namespace: openshift-logging
                tls:
                  ca:
                    key: service-ca.crt
                    configMapName: openshift-service-ca.crt
            pipelines:
              - name: my-syslog
                inputRefs:
                  - syslog-receiver
                outputRefs:
                  - default-lokistack 
      1. Watch collector pod.

      Actual results:

      Pod takes a lot of time between the creation event and the actual networking attachment and start working:

      Events:
        Type     Reason          Age                  From               Message
        ----     ------          ----                 ----               -------
        Normal   Scheduled       48m                  default-scheduler  Successfully assigned openshift-logging/collector-xhgxw to crc
        Normal   AddedInterface  38m                  multus             Add eth0 [10.217.0.69/23] from ovn-kubernetes
        Normal   Pulling         38m                  kubelet            Pulling image "registry.redhat.io/openshift-logging/vector-rhel9@sha256:cdb377d7a5140bfbad47681ed476c4d3138400c475c4fc7a36d77a3873710659"
        Normal   Pulled          38m                  kubelet            Successfully pulled image "registry.redhat.io/openshift-logging/vector-rhel9@sha256:cdb377d7a5140bfbad47681ed476c4d3138400c475c4fc7a36d77a3873710659" in 8.894s (8.894s including waiting)
        Normal   Created         38m                  kubelet            Created container collector
        Normal   Started         38m                  kubelet            Started container collector
        Warning  FailedMount     115s (x66 over 48m)  kubelet            MountVolume.SetUp failed for volume "collector-syslog-receiver" : secret "collector-syslog-receiver" not found 

      In this case, it took 10 minutes.

      Expected results:

      The collector pod starts almost immediately:

      Events:
        Type     Reason          Age                  From               Message
        ----     ------          ----                 ----               -------
        Normal   Scheduled       48m                  default-scheduler  Successfully assigned openshift-logging/collector-xhgxw to crc
        Normal   AddedInterface  48m                  multus             Add eth0 [10.217.0.69/23] from ovn-kubernetes
        Normal   Pulling         48m                  kubelet            Pulling image "registry.redhat.io/openshift-logging/vector-rhel9@sha256:cdb377d7a5140bfbad47681ed476c4d3138400c475c4fc7a36d77a3873710659"
        Normal   Pulled          48m                  kubelet            Successfully pulled image "registry.redhat.io/openshift-logging/vector-rhel9@sha256:cdb377d7a5140bfbad47681ed476c4d3138400c475c4fc7a36d77a3873710659" in 8.894s (8.894s including waiting)
        Normal   Created         48m                  kubelet            Created container collector
        Normal   Started         48m                  kubelet            Started container collector
        Warning  FailedMount     115s (x66 over 48m)  kubelet            MountVolume.SetUp failed for volume "collector-syslog-receiver" : secret "collector-syslog-receiver" not found 

      Additional info:

              Unassigned Unassigned
              rhn-engineering-jlarriba Juan Larriba
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: