Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-42479

[Dev]PCP pmcd.log fills /var filesystem with too many open files messages

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • rhel-8.4.0
    • pcp
    • None
    • None
    • 1
    • sst_pt_pcp
    • ssg_platform_tools
    • 2
    • None
    • PCP Sprint 5
    • None
    • None
    • None

      What were you trying to do that didn't work?

      The performance copilot(pmcd) and haproxy-pmcd and pmproxy services are installed on virtual machine to fetch the cluster details, 

      It is observed that over a period of time the logs of pmcd / pmproxy are increasing to huge size in path /var/log/pcp/ which eventually is leading to /var getting to 100% full

      If /var gets to 100% full, the risk is sudo to root on VM is not possible - which eventually means any critical administrative operations on server cannot be performed - this is an critical impact to SAP source system - since the SPOF (single point of faiure) SAP critical resources Message server and Database runs on these VM's

      After reading https://bugzilla.redhat.com/show_bug.cgi?id=1954711 we upgraded to the following:

      [root@amedbvm1 pmcd]# rpm -qa | grep pcp
      pcp-pmda-hacluster-5.3.5-8.el8.x86_64
      pcp-libs-5.3.5-8.el8.x86_64
      pcp-conf-5.3.5-8.el8.x86_64
      pcp-selinux-5.3.5-8.el8.x86_64
      pcp-5.3.5-8.el8.x86_64

      This seems to have resolved the issue with the pmproxy.log getting the "too many open files". but now we are seeing pmcd.log with a similar error.

      Please provide the package NVR for which bug is seen:

      How reproducible:

      This is ongoing in multiple systems

      Steps to reproduce

      1. 1. sudo yum install pcp pcp-pmda-hacluster

      2. sudo systemctl start pmcd

      3. sudo systemctl enable pmcd

              • HA cluster PMDA Install

      4. cd /var/lib/pcp/pmdas/hacluster

      5. sudo ./Install

      ***Start pmproxy

      6. sudo systemctl start pmproxy

      7. sudo systemctl enable pmproxy

      Expected results

      normal disk usage for the system

      Actual results

      either pmproxy.log or pmcd.log fills up the /var filesystem

            nathans@redhat.com Nathan Scott
            ross-msft Ross Sponholtz
            Microsoft Confidential Group
            pcp-maint pcp-maint
            Jan Kurik Jan Kurik
            Jacob Valdez Jacob Valdez (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: