-
Bug
-
Resolution: Done
-
Minor
-
Logging 5.5.0
-
False
-
None
-
False
-
NEW
-
VERIFIED
-
-
Log Collection - Sprint 226
Description of problem:
The ds/collector was created after creating clusterlogging, but a few seconds later, it's removed:
$ oc get pod NAME READY STATUS RESTARTS AGE cluster-logging-operator-749d8745d8-77cx7 1/1 Running 0 44s collector-55bkg 0/2 Terminating 0 7s collector-65dcq 0/2 Terminating 0 7s collector-6fdsb 0/2 Terminating 0 7s collector-h2kj7 0/2 Terminating 0 7s collector-k957n 0/2 Terminating 0 7s collector-mz8j4 0/2 Terminating 0 7s elasticsearch-cdm-pty5j1eb-1-778959d698-9jkq9 0/2 Pending 0 5s elasticsearch-cdm-pty5j1eb-2-57f5d964fc-g745c 0/2 Pending 0 4s elasticsearch-cdm-pty5j1eb-3-76c4f47596-pdfnp 0/2 Pending 0 3s $ oc get ds No resources found in openshift-logging namespace. $ oc get cm NAME DATA AGE elasticsearch 3 11s indexmanagement-scripts 9 8s kibana-trusted-ca-bundle 1 4s kube-root-ca.crt 1 64s openshift-service-ca.crt 1 64s $ oc describe pod collector-55bkg Name: collector-55bkg ...... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 28s default-scheduler Successfully assigned openshift-logging/collector-55bkg to ip-10-0-169-120.us-east-2.compute.internal by ip-10-0-169-120 Warning FailedMount 27s kubelet MountVolume.SetUp failed for volume "collector" : secret "collector" not found Warning FailedMount 12s (x6 over 27s) kubelet MountVolume.SetUp failed for volume "collector-metrics" : secret "collector-metrics" not found Warning FailedMount 12s (x6 over 27s) kubelet MountVolume.SetUp failed for volume "config" : configmap "collector" not found Warning FailedMount 12s (x6 over 27s) kubelet MountVolume.SetUp failed for volume "entrypoint" : configmap "collector" not found Warning FailedMount 11s (x6 over 27s) kubelet MountVolume.SetUp failed for volume "kube-api-access-mb6lc" : failed to fetch token: serviceaccounts "logcollector" not found
then it's created again, but it's removed soon:
$ oc get pod NAME READY STATUS RESTARTS AGE cluster-logging-operator-749d8745d8-77cx7 1/1 Running 0 89s collector-55bkg 0/2 Terminating 0 52s collector-65dcq 0/2 Terminating 0 52s collector-6fdsb 0/2 Terminating 0 52s collector-8j55g 0/2 Terminating 0 3s collector-fbkn9 0/2 Terminating 0 3s collector-fknwj 0/2 Terminating 0 3s collector-h2kj7 0/2 Terminating 0 52s collector-k957n 0/2 Terminating 0 52s collector-mz8j4 0/2 Terminating 0 52s collector-r4c5t 0/2 Terminating 0 3s collector-sw58q 0/2 Terminating 0 3s collector-tdbxx 0/2 Terminating 0 3s elasticsearch-cdm-pty5j1eb-1-778959d698-9jkq9 1/2 Running 0 50s elasticsearch-cdm-pty5j1eb-2-57f5d964fc-g745c 1/2 Running 0 49s elasticsearch-cdm-pty5j1eb-3-76c4f47596-pdfnp 1/2 Running 0 48s kibana-774fb8ddc6-94zck 2/2 Running 0 43s [qitang@preserve-docker-slave aws]$ oc get ds No resources found in openshift-logging namespace. [qitang@preserve-docker-slave aws]$ oc describe pod collector-8j55g Name: collector-8j55g Namespace: openshift-logging ...... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 18s default-scheduler Successfully assigned openshift-logging/collector-8j55g to ip-10-0-169-120.us-east-2.compute.internal by ip-10-0-169-120 Warning FailedMount 2s (x6 over 18s) kubelet MountVolume.SetUp failed for volume "collector-metrics" : secret "collector-metrics" not found Warning FailedMount 2s (x6 over 18s) kubelet MountVolume.SetUp failed for volume "entrypoint" : configmap "collector" not found Warning FailedMount 2s (x6 over 18s) kubelet MountVolume.SetUp failed for volume "config" : configmap "collector" not found Warning FailedMount 2s (x6 over 18s) kubelet MountVolume.SetUp failed for volume "kube-api-access-lnf28" : failed to fetch token: serviceaccounts "logcollector" not found
and another 1 or 2 minutes later, it's created.
Events in openshift-logging:
$ oc get events |grep collector 11m Normal Scheduled pod/collector-527nb Successfully assigned openshift-logging/collector-527nb to ip-10-0-184-249.us-east-2.compute.internal by ip-10-0-169-120 11m Normal AddedInterface pod/collector-527nb Add eth0 [10.129.2.236/23] from openshift-sdn 11m Normal Pulled pod/collector-527nb Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 11m Normal Created pod/collector-527nb Created container collector 11m Normal Started pod/collector-527nb Started container collector 11m Normal Pulled pod/collector-527nb Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 11m Normal Created pod/collector-527nb Created container logfilesmetricexporter 11m Normal Started pod/collector-527nb Started container logfilesmetricexporter 10m Normal Killing pod/collector-527nb Stopping container collector 10m Normal Killing pod/collector-527nb Stopping container logfilesmetricexporter 13m Normal Scheduled pod/collector-55bkg Successfully assigned openshift-logging/collector-55bkg to ip-10-0-169-120.us-east-2.compute.internal by ip-10-0-169-120 13m Warning FailedMount pod/collector-55bkg MountVolume.SetUp failed for volume "collector" : secret "collector" not found 12m Warning FailedMount pod/collector-55bkg MountVolume.SetUp failed for volume "collector-metrics" : secret "collector-metrics" not found 12m Warning FailedMount pod/collector-55bkg MountVolume.SetUp failed for volume "config" : configmap "collector" not found 12m Warning FailedMount pod/collector-55bkg MountVolume.SetUp failed for volume "entrypoint" : configmap "collector" not found 12m Warning FailedMount pod/collector-55bkg MountVolume.SetUp failed for volume "kube-api-access-mb6lc" : failed to fetch token: serviceaccounts "logcollector" not found 13m Normal Scheduled pod/collector-65dcq Successfully assigned openshift-logging/collector-65dcq to ip-10-0-184-249.us-east-2.compute.internal by ip-10-0-169-120 12m Warning FailedMount pod/collector-65dcq MountVolume.SetUp failed for volume "config" : configmap "collector" not found 12m Warning FailedMount pod/collector-65dcq MountVolume.SetUp failed for volume "entrypoint" : configmap "collector" not found 13m Warning FailedMount pod/collector-65dcq MountVolume.SetUp failed for volume "collector" : secret "collector" not found 12m Warning FailedMount pod/collector-65dcq MountVolume.SetUp failed for volume "kube-api-access-4hppl" : failed to fetch token: serviceaccounts "logcollector" not found 13m Normal Scheduled pod/collector-6fdsb Successfully assigned openshift-logging/collector-6fdsb to ip-10-0-215-108.us-east-2.compute.internal by ip-10-0-169-120 12m Warning FailedMount pod/collector-6fdsb MountVolume.SetUp failed for volume "entrypoint" : configmap "collector" not found 12m Warning FailedMount pod/collector-6fdsb MountVolume.SetUp failed for volume "config" : configmap "collector" not found 12m Warning FailedMount pod/collector-6fdsb MountVolume.SetUp failed for volume "collector-metrics" : secret "collector-metrics" not found 13m Warning FailedMount pod/collector-6fdsb MountVolume.SetUp failed for volume "collector" : secret "collector" not found 12m Warning FailedMount pod/collector-6fdsb MountVolume.SetUp failed for volume "kube-api-access-mmszg" : failed to fetch token: serviceaccounts "logcollector" not found 11m Normal Scheduled pod/collector-84b8d Successfully assigned openshift-logging/collector-84b8d to ip-10-0-129-35.us-east-2.compute.internal by ip-10-0-169-120 11m Normal AddedInterface pod/collector-84b8d Add eth0 [10.128.2.231/23] from openshift-sdn 11m Normal Pulled pod/collector-84b8d Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 11m Normal Created pod/collector-84b8d Created container collector 11m Normal Started pod/collector-84b8d Started container collector 11m Normal Pulled pod/collector-84b8d Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 11m Normal Created pod/collector-84b8d Created container logfilesmetricexporter 11m Normal Started pod/collector-84b8d Started container logfilesmetricexporter 10m Normal Killing pod/collector-84b8d Stopping container collector 10m Normal Killing pod/collector-84b8d Stopping container logfilesmetricexporter 12m Normal Scheduled pod/collector-8j55g Successfully assigned openshift-logging/collector-8j55g to ip-10-0-169-120.us-east-2.compute.internal by ip-10-0-169-120 12m Warning FailedMount pod/collector-8j55g MountVolume.SetUp failed for volume "collector-metrics" : secret "collector-metrics" not found 12m Warning FailedMount pod/collector-8j55g MountVolume.SetUp failed for volume "entrypoint" : configmap "collector" not found 12m Warning FailedMount pod/collector-8j55g MountVolume.SetUp failed for volume "config" : configmap "collector" not found 12m Warning FailedMount pod/collector-8j55g MountVolume.SetUp failed for volume "kube-api-access-lnf28" : failed to fetch token: serviceaccounts "logcollector" not found 11m Normal AddedInterface pod/collector-8j55g Add eth0 [10.129.0.165/23] from openshift-sdn 11m Normal Pulled pod/collector-8j55g Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 10m Normal Scheduled pod/collector-8rmv6 Successfully assigned openshift-logging/collector-8rmv6 to ip-10-0-184-249.us-east-2.compute.internal by ip-10-0-169-120 10m Normal AddedInterface pod/collector-8rmv6 Add eth0 [10.129.2.237/23] from openshift-sdn 10m Normal Pulled pod/collector-8rmv6 Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 10m Normal Created pod/collector-8rmv6 Created container collector 10m Normal Started pod/collector-8rmv6 Started container collector 10m Normal Pulled pod/collector-8rmv6 Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 10m Normal Created pod/collector-8rmv6 Created container logfilesmetricexporter 10m Normal Started pod/collector-8rmv6 Started container logfilesmetricexporter 11m Normal Scheduled pod/collector-bh57t Successfully assigned openshift-logging/collector-bh57t to ip-10-0-156-206.us-east-2.compute.internal by ip-10-0-169-120 11m Normal AddedInterface pod/collector-bh57t Add eth0 [10.130.0.185/23] from openshift-sdn 11m Normal Pulled pod/collector-bh57t Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 11m Normal Created pod/collector-bh57t Created container collector 11m Normal Started pod/collector-bh57t Started container collector 11m Normal Pulled pod/collector-bh57t Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 11m Normal Created pod/collector-bh57t Created container logfilesmetricexporter 11m Normal Started pod/collector-bh57t Started container logfilesmetricexporter 10m Normal Killing pod/collector-bh57t Stopping container collector 10m Normal Killing pod/collector-bh57t Stopping container logfilesmetricexporter 10m Normal Scheduled pod/collector-dsmr4 Successfully assigned openshift-logging/collector-dsmr4 to ip-10-0-219-34.us-east-2.compute.internal by ip-10-0-169-120 10m Normal AddedInterface pod/collector-dsmr4 Add eth0 [10.128.1.11/23] from openshift-sdn 10m Normal Pulled pod/collector-dsmr4 Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 10m Normal Created pod/collector-dsmr4 Created container collector 10m Normal Started pod/collector-dsmr4 Started container collector 10m Normal Pulled pod/collector-dsmr4 Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 10m Normal Created pod/collector-dsmr4 Created container logfilesmetricexporter 10m Normal Started pod/collector-dsmr4 Started container logfilesmetricexporter 12m Normal Scheduled pod/collector-fbkn9 Successfully assigned openshift-logging/collector-fbkn9 to ip-10-0-129-35.us-east-2.compute.internal by ip-10-0-169-120 12m Warning FailedMount pod/collector-fbkn9 MountVolume.SetUp failed for volume "config" : configmap "collector" not found 12m Warning FailedMount pod/collector-fbkn9 MountVolume.SetUp failed for volume "entrypoint" : configmap "collector" not found 12m Warning FailedMount pod/collector-fbkn9 MountVolume.SetUp failed for volume "collector-metrics" : secret "collector-metrics" not found 12m Warning FailedMount pod/collector-fbkn9 MountVolume.SetUp failed for volume "kube-api-access-mxhhv" : failed to fetch token: serviceaccounts "logcollector" not found 11m Normal AddedInterface pod/collector-fbkn9 Add eth0 [10.128.2.230/23] from openshift-sdn 11m Normal Pulled pod/collector-fbkn9 Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 12m Normal Scheduled pod/collector-fknwj Successfully assigned openshift-logging/collector-fknwj to ip-10-0-156-206.us-east-2.compute.internal by ip-10-0-169-120 12m Warning FailedMount pod/collector-fknwj MountVolume.SetUp failed for volume "collector-metrics" : secret "collector-metrics" not found 11m Normal AddedInterface pod/collector-fknwj Add eth0 [10.130.0.184/23] from openshift-sdn 11m Normal Pulled pod/collector-fknwj Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 11m Normal Created pod/collector-fknwj Created container collector 11m Normal Started pod/collector-fknwj Started container collector 11m Normal Pulled pod/collector-fknwj Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 11m Normal Created pod/collector-fknwj Created container logfilesmetricexporter 11m Normal Started pod/collector-fknwj Started container logfilesmetricexporter 11m Normal Killing pod/collector-fknwj Stopping container collector 11m Normal Killing pod/collector-fknwj Stopping container logfilesmetricexporter 13m Normal Scheduled pod/collector-h2kj7 Successfully assigned openshift-logging/collector-h2kj7 to ip-10-0-219-34.us-east-2.compute.internal by ip-10-0-169-120 12m Warning FailedMount pod/collector-h2kj7 MountVolume.SetUp failed for volume "config" : configmap "collector" not found 12m Warning FailedMount pod/collector-h2kj7 MountVolume.SetUp failed for volume "entrypoint" : configmap "collector" not found 13m Warning FailedMount pod/collector-h2kj7 MountVolume.SetUp failed for volume "collector" : secret "collector" not found 12m Warning FailedMount pod/collector-h2kj7 MountVolume.SetUp failed for volume "kube-api-access-xrl4g" : failed to fetch token: serviceaccounts "logcollector" not found 13m Normal Scheduled pod/collector-k957n Successfully assigned openshift-logging/collector-k957n to ip-10-0-129-35.us-east-2.compute.internal by ip-10-0-169-120 12m Warning FailedMount pod/collector-k957n MountVolume.SetUp failed for volume "config" : configmap "collector" not found 12m Warning FailedMount pod/collector-k957n MountVolume.SetUp failed for volume "entrypoint" : configmap "collector" not found 13m Warning FailedMount pod/collector-k957n MountVolume.SetUp failed for volume "collector" : secret "collector" not found 12m Warning FailedMount pod/collector-k957n MountVolume.SetUp failed for volume "collector-metrics" : secret "collector-metrics" not found 12m Warning FailedMount pod/collector-k957n MountVolume.SetUp failed for volume "kube-api-access-z4fjb" : failed to fetch token: serviceaccounts "logcollector" not found 11m Normal Scheduled pod/collector-kwwmw Successfully assigned openshift-logging/collector-kwwmw to ip-10-0-219-34.us-east-2.compute.internal by ip-10-0-169-120 11m Normal AddedInterface pod/collector-kwwmw Add eth0 [10.128.1.7/23] from openshift-sdn 11m Normal Pulled pod/collector-kwwmw Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 11m Normal Created pod/collector-kwwmw Created container collector 11m Normal Started pod/collector-kwwmw Started container collector 11m Normal Pulled pod/collector-kwwmw Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 11m Normal Created pod/collector-kwwmw Created container logfilesmetricexporter 11m Normal Started pod/collector-kwwmw Started container logfilesmetricexporter 11m Normal Killing pod/collector-kwwmw Stopping container collector 11m Normal Killing pod/collector-kwwmw Stopping container logfilesmetricexporter 11m Normal Scheduled pod/collector-mskq2 Successfully assigned openshift-logging/collector-mskq2 to ip-10-0-169-120.us-east-2.compute.internal by ip-10-0-169-120 11m Warning FailedMount pod/collector-mskq2 MountVolume.SetUp failed for volume "collector-metrics" : secret "collector-metrics" not found 11m Normal AddedInterface pod/collector-mskq2 Add eth0 [10.129.0.166/23] from openshift-sdn 11m Normal Pulled pod/collector-mskq2 Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 11m Normal Created pod/collector-mskq2 Created container collector 11m Normal Started pod/collector-mskq2 Started container collector 11m Normal Pulled pod/collector-mskq2 Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 11m Normal Created pod/collector-mskq2 Created container logfilesmetricexporter 11m Normal Started pod/collector-mskq2 Started container logfilesmetricexporter 10m Normal Killing pod/collector-mskq2 Stopping container collector 10m Normal Killing pod/collector-mskq2 Stopping container logfilesmetricexporter 13m Normal Scheduled pod/collector-mz8j4 Successfully assigned openshift-logging/collector-mz8j4 to ip-10-0-156-206.us-east-2.compute.internal by ip-10-0-169-120 12m Warning FailedMount pod/collector-mz8j4 MountVolume.SetUp failed for volume "collector-metrics" : secret "collector-metrics" not found 12m Warning FailedMount pod/collector-mz8j4 MountVolume.SetUp failed for volume "config" : configmap "collector" not found 12m Warning FailedMount pod/collector-mz8j4 MountVolume.SetUp failed for volume "entrypoint" : configmap "collector" not found 13m Warning FailedMount pod/collector-mz8j4 MountVolume.SetUp failed for volume "collector" : secret "collector" not found 12m Warning FailedMount pod/collector-mz8j4 MountVolume.SetUp failed for volume "kube-api-access-7cwwx" : failed to fetch token: serviceaccounts "logcollector" not found 11m Normal Scheduled pod/collector-p6vft Successfully assigned openshift-logging/collector-p6vft to ip-10-0-215-108.us-east-2.compute.internal by ip-10-0-169-120 11m Normal AddedInterface pod/collector-p6vft Add eth0 [10.131.0.182/23] from openshift-sdn 11m Normal Pulled pod/collector-p6vft Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 11m Normal Created pod/collector-p6vft Created container collector 11m Normal Started pod/collector-p6vft Started container collector 11m Normal Pulled pod/collector-p6vft Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 11m Normal Created pod/collector-p6vft Created container logfilesmetricexporter 11m Normal Started pod/collector-p6vft Started container logfilesmetricexporter 11m Normal Killing pod/collector-p6vft Stopping container collector 11m Normal Killing pod/collector-p6vft Stopping container logfilesmetricexporter 11m Normal Scheduled pod/collector-qnt8j Successfully assigned openshift-logging/collector-qnt8j to ip-10-0-156-206.us-east-2.compute.internal by ip-10-0-169-120 11m Normal AddedInterface pod/collector-qnt8j Add eth0 [10.130.0.183/23] from openshift-sdn 11m Normal Pulled pod/collector-qnt8j Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 11m Normal Created pod/collector-qnt8j Created container collector 11m Normal Started pod/collector-qnt8j Started container collector 11m Normal Pulled pod/collector-qnt8j Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 11m Normal Created pod/collector-qnt8j Created container logfilesmetricexporter 11m Normal Started pod/collector-qnt8j Started container logfilesmetricexporter 11m Normal Killing pod/collector-qnt8j Stopping container collector 11m Normal Killing pod/collector-qnt8j Stopping container logfilesmetricexporter 10m Normal Scheduled pod/collector-qrd77 Successfully assigned openshift-logging/collector-qrd77 to ip-10-0-156-206.us-east-2.compute.internal by ip-10-0-169-120 10m Normal AddedInterface pod/collector-qrd77 Add eth0 [10.130.0.186/23] from openshift-sdn 10m Normal Pulled pod/collector-qrd77 Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 10m Normal Created pod/collector-qrd77 Created container collector 10m Normal Started pod/collector-qrd77 Started container collector 10m Normal Pulled pod/collector-qrd77 Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 10m Normal Created pod/collector-qrd77 Created container logfilesmetricexporter 10m Normal Started pod/collector-qrd77 Started container logfilesmetricexporter 12m Normal Scheduled pod/collector-r4c5t Successfully assigned openshift-logging/collector-r4c5t to ip-10-0-184-249.us-east-2.compute.internal by ip-10-0-169-120 12m Warning FailedMount pod/collector-r4c5t MountVolume.SetUp failed for volume "collector-metrics" : secret "collector-metrics" not found 12m Warning FailedMount pod/collector-r4c5t MountVolume.SetUp failed for volume "config" : configmap "collector" not found 12m Warning FailedMount pod/collector-r4c5t MountVolume.SetUp failed for volume "entrypoint" : configmap "collector" not found 12m Warning FailedMount pod/collector-r4c5t MountVolume.SetUp failed for volume "kube-api-access-rbd58" : failed to fetch token: serviceaccounts "logcollector" not found 11m Normal AddedInterface pod/collector-r4c5t Add eth0 [10.129.2.235/23] from openshift-sdn 11m Normal Pulled pod/collector-r4c5t Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 11m Normal Scheduled pod/collector-r6h8l Successfully assigned openshift-logging/collector-r6h8l to ip-10-0-215-108.us-east-2.compute.internal by ip-10-0-169-120 11m Normal AddedInterface pod/collector-r6h8l Add eth0 [10.131.0.184/23] from openshift-sdn 11m Normal Pulled pod/collector-r6h8l Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 11m Normal Created pod/collector-r6h8l Created container collector 11m Normal Started pod/collector-r6h8l Started container collector 11m Normal Pulled pod/collector-r6h8l Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 11m Normal Created pod/collector-r6h8l Created container logfilesmetricexporter 11m Normal Started pod/collector-r6h8l Started container logfilesmetricexporter 10m Normal Killing pod/collector-r6h8l Stopping container collector 10m Normal Killing pod/collector-r6h8l Stopping container logfilesmetricexporter 12m Normal Scheduled pod/collector-sw58q Successfully assigned openshift-logging/collector-sw58q to ip-10-0-219-34.us-east-2.compute.internal by ip-10-0-169-120 12m Warning FailedMount pod/collector-sw58q MountVolume.SetUp failed for volume "collector-metrics" : secret "collector-metrics" not found 11m Normal AddedInterface pod/collector-sw58q Add eth0 [10.128.1.9/23] from openshift-sdn 11m Normal Pulled pod/collector-sw58q Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 11m Normal Created pod/collector-sw58q Created container collector 11m Normal Started pod/collector-sw58q Started container collector 11m Normal Pulled pod/collector-sw58q Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 11m Normal Created pod/collector-sw58q Created container logfilesmetricexporter 11m Normal Started pod/collector-sw58q Started container logfilesmetricexporter 11m Normal Killing pod/collector-sw58q Stopping container collector 11m Normal Killing pod/collector-sw58q Stopping container logfilesmetricexporter 12m Normal Scheduled pod/collector-tdbxx Successfully assigned openshift-logging/collector-tdbxx to ip-10-0-215-108.us-east-2.compute.internal by ip-10-0-169-120 12m Warning FailedMount pod/collector-tdbxx MountVolume.SetUp failed for volume "entrypoint" : configmap "collector" not found 12m Warning FailedMount pod/collector-tdbxx MountVolume.SetUp failed for volume "config" : configmap "collector" not found 12m Warning FailedMount pod/collector-tdbxx MountVolume.SetUp failed for volume "collector-metrics" : secret "collector-metrics" not found 12m Warning FailedMount pod/collector-tdbxx MountVolume.SetUp failed for volume "kube-api-access-kxz9c" : failed to fetch token: serviceaccounts "logcollector" not found 11m Normal AddedInterface pod/collector-tdbxx Add eth0 [10.131.0.183/23] from openshift-sdn 11m Normal Pulled pod/collector-tdbxx Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 10m Normal Scheduled pod/collector-ttq4p Successfully assigned openshift-logging/collector-ttq4p to ip-10-0-215-108.us-east-2.compute.internal by ip-10-0-169-120 10m Normal AddedInterface pod/collector-ttq4p Add eth0 [10.131.0.186/23] from openshift-sdn 10m Normal Pulled pod/collector-ttq4p Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 10m Normal Created pod/collector-ttq4p Created container collector 10m Normal Started pod/collector-ttq4p Started container collector 10m Normal Pulled pod/collector-ttq4p Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 10m Normal Created pod/collector-ttq4p Created container logfilesmetricexporter 10m Normal Started pod/collector-ttq4p Started container logfilesmetricexporter 10m Normal Scheduled pod/collector-tzrkh Successfully assigned openshift-logging/collector-tzrkh to ip-10-0-169-120.us-east-2.compute.internal by ip-10-0-169-120 10m Normal AddedInterface pod/collector-tzrkh Add eth0 [10.129.0.167/23] from openshift-sdn 10m Normal Pulled pod/collector-tzrkh Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 10m Normal Created pod/collector-tzrkh Created container collector 10m Normal Started pod/collector-tzrkh Started container collector 10m Normal Pulled pod/collector-tzrkh Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 10m Normal Created pod/collector-tzrkh Created container logfilesmetricexporter 10m Normal Started pod/collector-tzrkh Started container logfilesmetricexporter 11m Normal Scheduled pod/collector-v2x2d Successfully assigned openshift-logging/collector-v2x2d to ip-10-0-169-120.us-east-2.compute.internal by ip-10-0-169-120 11m Normal AddedInterface pod/collector-v2x2d Add eth0 [10.129.0.164/23] from openshift-sdn 11m Normal Pulled pod/collector-v2x2d Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 11m Normal Created pod/collector-v2x2d Created container collector 11m Normal Started pod/collector-v2x2d Started container collector 11m Normal Pulled pod/collector-v2x2d Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 11m Normal Created pod/collector-v2x2d Created container logfilesmetricexporter 11m Normal Started pod/collector-v2x2d Started container logfilesmetricexporter 11m Normal Killing pod/collector-v2x2d Stopping container collector 11m Normal Killing pod/collector-v2x2d Stopping container logfilesmetricexporter 11m Normal Scheduled pod/collector-vdrz4 Successfully assigned openshift-logging/collector-vdrz4 to ip-10-0-129-35.us-east-2.compute.internal by ip-10-0-169-120 11m Normal AddedInterface pod/collector-vdrz4 Add eth0 [10.128.2.229/23] from openshift-sdn 11m Normal Pulled pod/collector-vdrz4 Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 11m Normal Created pod/collector-vdrz4 Created container collector 11m Normal Started pod/collector-vdrz4 Started container collector 11m Normal Pulled pod/collector-vdrz4 Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 11m Normal Created pod/collector-vdrz4 Created container logfilesmetricexporter 11m Normal Started pod/collector-vdrz4 Started container logfilesmetricexporter 11m Normal Killing pod/collector-vdrz4 Stopping container collector 11m Normal Killing pod/collector-vdrz4 Stopping container logfilesmetricexporter 10m Normal Scheduled pod/collector-xjkb7 Successfully assigned openshift-logging/collector-xjkb7 to ip-10-0-129-35.us-east-2.compute.internal by ip-10-0-169-120 10m Normal AddedInterface pod/collector-xjkb7 Add eth0 [10.128.2.235/23] from openshift-sdn 10m Normal Pulled pod/collector-xjkb7 Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 10m Normal Created pod/collector-xjkb7 Created container collector 10m Normal Started pod/collector-xjkb7 Started container collector 10m Normal Pulled pod/collector-xjkb7 Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 10m Normal Created pod/collector-xjkb7 Created container logfilesmetricexporter 10m Normal Started pod/collector-xjkb7 Started container logfilesmetricexporter 11m Normal Scheduled pod/collector-z827m Successfully assigned openshift-logging/collector-z827m to ip-10-0-219-34.us-east-2.compute.internal by ip-10-0-169-120 11m Warning FailedMount pod/collector-z827m MountVolume.SetUp failed for volume "collector-metrics" : secret "collector-metrics" not found 11m Normal AddedInterface pod/collector-z827m Add eth0 [10.128.1.10/23] from openshift-sdn 11m Normal Pulled pod/collector-z827m Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 11m Normal Created pod/collector-z827m Created container collector 11m Normal Started pod/collector-z827m Started container collector 11m Normal Pulled pod/collector-z827m Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 11m Normal Created pod/collector-z827m Created container logfilesmetricexporter 11m Normal Started pod/collector-z827m Started container logfilesmetricexporter 10m Normal Killing pod/collector-z827m Stopping container collector 10m Normal Killing pod/collector-z827m Stopping container logfilesmetricexporter 11m Normal Scheduled pod/collector-z8qkl Successfully assigned openshift-logging/collector-z8qkl to ip-10-0-184-249.us-east-2.compute.internal by ip-10-0-169-120 11m Normal AddedInterface pod/collector-z8qkl Add eth0 [10.129.2.234/23] from openshift-sdn 11m Normal Pulled pod/collector-z8qkl Container image "registry.redhat.io/openshift-logging/fluentd-rhel8@sha256:9c98213dee72cf1877238791200f299e3c62367d5aeb141b2c39236c8d70de4b" already present on machine 11m Normal Created pod/collector-z8qkl Created container collector 11m Normal Started pod/collector-z8qkl Started container collector 11m Normal Pulled pod/collector-z8qkl Container image "registry.redhat.io/openshift-logging/log-file-metric-exporter-rhel8@sha256:c9fe5a75a9d38e95edbefd84ff91cdab9f547f9588e909d138c43daf332dcccb" already present on machine 11m Normal Created pod/collector-z8qkl Created container logfilesmetricexporter 11m Normal Started pod/collector-z8qkl Started container logfilesmetricexporter 11m Normal Killing pod/collector-z8qkl Stopping container collector 11m Normal Killing pod/collector-z8qkl Stopping container logfilesmetricexporter 13m Normal SuccessfulCreate daemonset/collector Created pod: collector-6fdsb 13m Normal SuccessfulCreate daemonset/collector Created pod: collector-h2kj7 13m Normal SuccessfulCreate daemonset/collector Created pod: collector-k957n 13m Normal SuccessfulCreate daemonset/collector Created pod: collector-65dcq 13m Normal SuccessfulCreate daemonset/collector Created pod: collector-55bkg 13m Normal SuccessfulCreate daemonset/collector Created pod: collector-mz8j4 12m Normal SuccessfulCreate daemonset/collector Created pod: collector-sw58q 12m Normal SuccessfulCreate daemonset/collector Created pod: collector-fknwj 12m Normal SuccessfulCreate daemonset/collector Created pod: collector-fbkn9 12m Normal SuccessfulCreate daemonset/collector Created pod: collector-r4c5t 12m Normal SuccessfulCreate daemonset/collector Created pod: collector-8j55g 12m Normal SuccessfulCreate daemonset/collector Created pod: collector-tdbxx 11m Normal SuccessfulCreate daemonset/collector Created pod: collector-v2x2d 11m Normal SuccessfulCreate daemonset/collector Created pod: collector-z8qkl 11m Normal SuccessfulCreate daemonset/collector Created pod: collector-p6vft 11m Normal SuccessfulCreate daemonset/collector Created pod: collector-kwwmw 11m Normal SuccessfulCreate daemonset/collector Created pod: collector-vdrz4 11m Normal SuccessfulCreate daemonset/collector Created pod: collector-qnt8j 11m Normal SuccessfulDelete daemonset/collector Deleted pod: collector-vdrz4 11m Normal SuccessfulDelete daemonset/collector Deleted pod: collector-z8qkl 11m Normal SuccessfulDelete daemonset/collector Deleted pod: collector-p6vft 11m Normal SuccessfulDelete daemonset/collector Deleted pod: collector-kwwmw 11m Normal SuccessfulDelete daemonset/collector Deleted pod: collector-qnt8j 11m Normal SuccessfulDelete daemonset/collector Deleted pod: collector-v2x2d 11m Normal SuccessfulCreate daemonset/collector Created pod: collector-z827m 11m Normal SuccessfulCreate daemonset/collector Created pod: collector-84b8d 11m Normal SuccessfulCreate daemonset/collector Created pod: collector-bh57t 11m Normal SuccessfulCreate daemonset/collector Created pod: collector-527nb 11m Normal SuccessfulCreate daemonset/collector Created pod: collector-mskq2 11m Normal SuccessfulCreate daemonset/collector Created pod: collector-r6h8l 10m Normal SuccessfulDelete daemonset/collector Deleted pod: collector-z827m 10m Normal SuccessfulDelete daemonset/collector Deleted pod: collector-mskq2 10m Normal SuccessfulDelete daemonset/collector Deleted pod: collector-84b8d 10m Normal SuccessfulDelete daemonset/collector Deleted pod: collector-r6h8l 10m Normal SuccessfulDelete daemonset/collector Deleted pod: collector-527nb 10m Normal SuccessfulDelete daemonset/collector Deleted pod: collector-bh57t 10m Warning FailedToUpdateEndpoint endpoints/collector Failed to update endpoint openshift-logging/collector: Operation cannot be fulfilled on endpoints "collector": the object has been modified; please apply your changes to the latest version and try again 10m Normal SuccessfulCreate daemonset/collector Created pod: collector-qrd77 10m Normal SuccessfulCreate daemonset/collector Created pod: collector-xjkb7 10m Normal SuccessfulCreate daemonset/collector Created pod: collector-ttq4p 10m Normal SuccessfulCreate daemonset/collector (combined from similar events): Created pod: collector-8rmv6
Version-Release number of selected component (if applicable):
clusterlogging.v5.5.0
How reproducible:
Always
Steps to Reproduce:
1. subscribe CLO and EO
2. create clusterlogging with:
apiVersion: "logging.openshift.io/v1" kind: "ClusterLogging" metadata: name: "instance" spec: managementState: "Managed" logStore: type: "elasticsearch" retentionPolicy: application: maxAge: 12h infra: maxAge: 12h audit: maxAge: 1d elasticsearch: nodeCount: 3 redundancyPolicy: "SingleRedundancy" resources: requests: memory: "2Gi" storage: storageClassName: "gp2" size: "20Gi" visualization: type: "kibana" kibana: resources: {} replicas: 1 collection: logs: type: fluentd fluentd: {}
3. monitor the ds/collector status
Actual results:
Expected results:
The ds/collector shouldn't be removed
Additional info:
- links to
- mentioned on
