-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
rhel-9.6
-
No
-
Important
-
1
-
rhel-pt-pcp
-
5
-
QE ack
-
False
-
False
-
-
Yes
-
Red Hat Enterprise Linux
-
PT PCP Splitted
-
None
-
Automated
-
Unspecified Release Note Type - Unknown
-
Unspecified
-
Unspecified
-
Unspecified
-
None
When querying pmproxy using HTTP, we can see a high memory leak occurring:
# for i in $(seq 1 100); do curl -q http://127.0.0.1:44322/metrics -o /dev/null; done
Result while monitoring pmproxy memory consumption (sixth field is RSS):
# while :; do ps auxwwwm | grep pmproxy | grep -v grep; sleep 10; done pcp 8077 0.0 0.3 184544 13824 ? - 11:12 0:00 /usr/libexec/pcp/bin/pmproxy -F -A -i 127.0.0.1 pcp 8077 0.0 0.8 315616 30080 ? - 11:12 0:00 /usr/libexec/pcp/bin/pmproxy -F -A -i 127.0.0.1 pcp 8077 0.1 1.1 315616 42624 ? - 11:12 0:01 /usr/libexec/pcp/bin/pmproxy -F -A -i 127.0.0.1 pcp 8077 0.2 1.2 315616 48512 ? - 11:12 0:02 /usr/libexec/pcp/bin/pmproxy -F -A -i 127.0.0.1 pcp 8077 0.6 2.6 315616 98432 ? - 11:12 0:05 /usr/libexec/pcp/bin/pmproxy -F -A -i 127.0.0.1 pcp 8077 0.9 3.3 315616 125952 ? - 11:12 0:08 /usr/libexec/pcp/bin/pmproxy -F -A -i 127.0.0.1 pcp 8077 1.2 4.2 315616 157312 ? - 11:12 0:12 /usr/libexec/pcp/bin/pmproxy -F -A -i 127.0.0.1 pcp 8077 1.6 5.0 315616 188928 ? - 11:12 0:15 /usr/libexec/pcp/bin/pmproxy -F -A -i 127.0.0.1 pcp 8077 1.8 5.6 315616 209920 ? - 11:12 0:18 /usr/libexec/pcp/bin/pmproxy -F -A -i 127.0.0.1 pcp 8077 1.8 5.6 315616 209920 ? - 11:12 0:18 /usr/libexec/pcp/bin/pmproxy -F -A -i 127.0.0.1 pcp 8077 1.8 5.6 315616 209920 ? - 11:12 0:18 /usr/libexec/pcp/bin/pmproxy -F -A -i 127.0.0.1 pcp 8077 1.7 5.6 315616 209920 ? - 11:12 0:18 /usr/libexec/pcp/bin/pmproxy -F -A -i 127.0.0.1
Executing pmproxy under valgrind shows memory is being allocated by on_pmwebapi_scrape():
476 static int 477 on_pmwebapi_scrape(sds context, pmWebScrape *scrape, void *arg) 478 { : 547 http_set_buffer(baton->client, result, HTTP_FLAG_TEXT); 548 http_transfer(baton->client); 549 return 0; 550 }
It seems that the result buffer is not always freed after being sent back to HTTP client, but I wasn't able to spot the location yet.
I'm attaching valgrind report, please check for "definitely lost" items.
Running under valgrind
# yum -y install yum-utils valgrind # systemctl start pmproxy # debuginfo-install -y $(rpm -qf $(awk '/\/usr\/lib64\// { print $6 }' /proc/$(pgrep pmproxy)/maps | sort -u) | sort -u) # mkdir /etc/systemd/system/pmproxy.service.d # cat > /etc/systemd/system/pmproxy.service.d/valgrind.conf << EOF [Service] ExecStart= ExecStart=/etc/pcp/pmproxy/rc.valgrind start-systemd EOF # sed '258s#exec #exec /usr/bin/valgrind --tool=memcheck --leak-check=full --trace-children=yes --track-origins=yes --log-file=/tmp/pmproxy.%p.log -- #' /etc/pcp/pmproxy/rc > /etc/pcp/pmproxy/rc.valgrind # chmod --reference=/etc/pcp/pmproxy/rc /etc/pcp/pmproxy/rc.valgrind # systemctl daemon-reload # systemctl restart pmproxy
- is cloned by
-
-
- Planning
-
- is related to
-
-
- Planning
-
