Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-58307

PCP pmcd.log fills /var filesystem with too many open files messages

    • pcp-6.2.2-7.el9_5
    • No
    • Moderate
    • ZStream
    • Customer Reported
    • 1
    • rhel-sst-pt-pcp
    • ssg_platform_tools
    • 2
    • QE ack, Dev ack
    • False
    • Hide

      None

      Show
      None
    • No
    • Red Hat Enterprise Linux
    • PCP Sprint 10
    • Release Note Not Required
    • x86_64
    • None

      What were you trying to do that didn't work?

      The performance copilot(pmcd) and haproxy-pmcd and pmproxy services are installed on virtual machine to fetch the cluster details, 

      It is observed that over a period of time the logs of pmcd / pmproxy are increasing to huge size in path /var/log/pcp/ which eventually is leading to /var getting to 100% full

      If /var gets to 100% full, the risk is sudo to root on VM is not possible - which eventually means any critical administrative operations on server cannot be performed - this is an critical impact to SAP source system - since the SPOF (single point of faiure) SAP critical resources Message server and Database runs on these VM's

      After reading https://bugzilla.redhat.com/show_bug.cgi?id=1954711 we upgraded to the following:

      [root@amedbvm1 pmcd]# rpm -qa | grep pcp
      pcp-pmda-hacluster-5.3.5-8.el8.x86_64
      pcp-libs-5.3.5-8.el8.x86_64
      pcp-conf-5.3.5-8.el8.x86_64
      pcp-selinux-5.3.5-8.el8.x86_64
      pcp-5.3.5-8.el8.x86_64

      This seems to have resolved the issue with the pmproxy.log getting the "too many open files". but now we are seeing pmcd.log with a similar error.

      Please provide the package NVR for which bug is seen:

      How reproducible:

      This is ongoing in multiple systems

      Steps to reproduce

      1. 1. sudo yum install pcp pcp-pmda-hacluster

      2. sudo systemctl start pmcd

      3. sudo systemctl enable pmcd

              • HA cluster PMDA Install

      4. cd /var/lib/pcp/pmdas/hacluster

      5. sudo ./Install

      ***Start pmproxy

      6. sudo systemctl start pmproxy

      7. sudo systemctl enable pmproxy

      Expected results

      normal disk usage for the system

      Actual results

      either pmproxy.log or pmcd.log fills up the /var filesystem

              nathans@redhat.com Nathan Scott
              ross-msft Ross Sponholtz (Inactive)
              Microsoft Confidential Group
              pcp-maint pcp-maint
              Jan Kurik Jan Kurik
              Jacob Valdez Jacob Valdez (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: