Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4395

Fluentd fails to get the kubernetes-metadata with error 404

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Normal Normal
    • None
    • Logging 5.7.z, Logging 5.6.z
    • Log Collection
    • False
    • None
    • False
    • NEW
    • NEW
    • Bug Fix
    • Moderate
    • Customer Escalated

      Description of problem:

      Fluentd errors observed getting the kubernetes-metadata:

       $ for pod in $(oc get pods -l component=collector -o name -n openshift-logging);do oc logs $pod -c collector -n openshift-logging|grep  "kubernetes-metadata";done |tail -2
2023-07-18T23:50:55.889790767Z 2023-07-18 23:50:55 +0000 [error]: [kubernetes-metadata] Exception 'HTTP status code 404, pods "pod1-78fd54d68c-58k7t" not found for GET https://kubernetes.default.svc/api/v1/namespaces/test/pods/pod1-78fd54d68c-58k7t' encountered fetching pod metadata from Kubernetes API v1 endpoint https://kubernetes.default.svc
2023-07-18T23:54:19.906137335Z 2023-07-18 23:54:19 +0000 [error]: [kubernetes-metadata] Exception 'HTTP status code 404, pods "pod2-78fd54d68c-jgwqp" not found for GET https://kubernetes.default.svc/api/v1/namespaces/test/pods/pod2-78fd54d68c-jgwqp' encountered fetching pod metadata from Kubernetes API v1 endpoint https://kubernetes.default.svc

      Even when it's not harmful, it's noisy in the fluentd pod logs.

      Version-Release number of selected component (if applicable):

      Cluster Logging Operator 5.6 using fluentd as collector

      How reproducible:

      No known exactly how to reproduce it, only knowing that the pods that fluentd is trying to get the metadata for pods that don't exist anymore.

      Steps to Reproduce:

      1.  
      2.  
      3. ...

      Actual results:

      Error thrown by fluentd as below:

       $ for pod in $(oc get pods -l component=collector -o name -n openshift-logging);do oc logs $pod -c collector -n openshift-logging|grep  "kubernetes-metadata";done |tail -2
2023-07-18T23:50:55.889790767Z 2023-07-18 23:50:55 +0000 [error]: [kubernetes-metadata] Exception 'HTTP status code 404, pods "pod1-78fd54d68c-58k7t" not found for GET https://kubernetes.default.svc/api/v1/namespaces/test/pods/pod1-78fd54d68c-58k7t' encountered fetching pod metadata from Kubernetes API v1 endpoint https://kubernetes.default.svc 2023-07-18T23:54:19.906137335Z 2023-07-18 23:54:19 +0000 [error]: [kubernetes-metadata] Exception 'HTTP status code 404, pods "pod2-78fd54d68c-jgwqp" not found for GET https://kubernetes.default.svc/api/v1/namespaces/test/pods/pod2-78fd54d68c-jgwqp' encountered fetching pod metadata from Kubernetes API v1 endpoint https://kubernetes.default.svc

      Expected results:

      Fluent is not trying to look for a pod that doesn't exist anymore and not error is observed

      Additional info:

      The fluentd pods are not exhausted in memory or cpu having enough resources

              Unassigned Unassigned
              rhn-support-ocasalsa Oscar Casal Sanchez
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: