Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2674

Many `can't remove non-existent inotify watch for: /var/log/pods/xxxxxx` errors in logfilesmetricexporter container.

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Log Collection - Sprint 224, Log Collection - Sprint 225

    Description

      Description of problem:

      logfilesmetricexporter container keeps reporting below errors:

      $ oc -n openshift-logging logs -c logfilesmetricexporter collector-m6cg6 -f
      {"component":"","level":"0","message":"start log metric exporter","path":"/var/log/pods","ts":"2022-05-31T02:56:21.998089878Z"}
      {"component":"","error":{"msg":"can't remove non-existent inotify watch for: /var/log/pods/openshift-logging_collector-kwzsg_3b3aaa34-8233-48ba-8f63-312f5258351f/collector"},"level":"0","message":"Error retrieving event","operation":"REMOVE","path":"/var/log/pods/openshift-logging_collector-kwzsg_3b3aaa34-8233-48ba-8f63-312f5258351f/collector","ts":"2022-05-31T02:56:26.384332941Z"}
      {"component":"","error":{"msg":"can't remove non-existent inotify watch for: /var/log/pods/openshift-logging_collector-kwzsg_3b3aaa34-8233-48ba-8f63-312f5258351f/collector"},"level":"0","message":"Error retrieving event","operation":"REMOVE","path":"/var/log/pods/openshift-logging_collector-kwzsg_3b3aaa34-8233-48ba-8f63-312f5258351f/collector","ts":"2022-05-31T02:56:26.384621621Z"}
      ......
      {"component":"","error":{"msg":"can't remove non-existent inotify watch for: /var/log/pods/openshift-logging_elasticsearch-im-infra-27566115-cv6mx_6dd07f39-2468-4926-91c0-f00a9145657d"},"level":"0","message":"Error retrieving event","operation":"REMOVE","path":"/var/log/pods/openshift-logging_elasticsearch-im-infra-27566115-cv6mx_6dd07f39-2468-4926-91c0-f00a9145657d","ts":"2022-05-31T03:30:26.896938222Z"}
      {"component":"","error":{"msg":"can't remove non-existent inotify watch for: /var/log/pods/openshift-marketplace_qe-app-registry-9jx5r_9f630faf-8516-4a4b-9b77-57e554fc03a4/registry-server/0.log"},"level":"0","message":"Error retrieving event","operation":"REMOVE","path":"/var/log/pods/openshift-marketplace_qe-app-registry-9jx5r_9f630faf-8516-4a4b-9b77-57e554fc03a4/registry-server/0.log","ts":"2022-05-31T03:31:45.275541262Z"}
      

       

      Version-Release number of selected component (if applicable):

      cluster-logging.5.4.2 

      How reproducible:

      Always

      Steps to Reproduce:

      1. deploy logging stack with:

      apiVersion: logging.openshift.io/v1
      kind: ClusterLogging
      metadata:
        name: instance
        namespace: openshift-logging
      spec:
        collection:
          logs:
            fluentd: {}
            type: fluentd
        logStore:
          elasticsearch:
            nodeCount: 1
            redundancyPolicy: ZeroRedundancy
            resources:
              requests:
                cpu: 100m
                memory: 1Gi
            storage:
              size: 20Gi
              storageClassName: standard
          retentionPolicy:
            application:
              maxAge: 60h
            audit:
              maxAge: 1d
            infra:
              maxAge: 3h
          type: elasticsearch
        managementState: Managed
        visualization:
          kibana:
            replicas: 1
          type: kibana 

      2. check logs in logfilesmetricexporter container

      Actual results:

      Expected results:

      No errors in the container logs

      Additional info:

      Attachments

        Activity

          People

            vparfono Vitalii Parfonov
            qitang@redhat.com Qiaoling Tang
            Qiaoling Tang Qiaoling Tang
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: