Normal Description
The Red Hat Logging Storage is overwhelmed by logs produced by the "comunity-operators" pod caused by a high log verbosity [0]
1.The stream rate limit in Loki for the "community-operators" pod is caused by a high log verbosity where thousands of messages are produced in the same second [0]
2.The most of the messages are related to the queries from the IP Addresses "10.128.0.25" and "10.129.0.19" [1]
3. The IP Addresses "10.128.0.25" and "10.129.0.19" are from the "packageserver" pods in the namespace "openshift-operator-lifecycle-manager" [2]
4. The "packageserver" pods trigger an update "PackageManifest" and the "community-operators" pod start to print all the stats for query methods "GetPackage|GetBundleForChannel|ListPackages|ListBundles" needed to update the "PackageManifest" [3]
Environment
BareMetal
$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.18.17 True False 5d Cluster version is 4.18.17
Observed behaviour
High log verbosity from the "community-operators" pods. These high verbosity impacts in all the Logging Stack and this at the same time impacts in the resources needed in the nodes, network, storage.
Expected behaviour
Reduce the log verbosity for the "community-operators" as the logs printed as more as "DEBUG" information or prone to be exposed as metrics and ingested by Prometheus to be analyzed later.
Workaround
Filter out the logs coming from the "community-operators" pod.
[0]
$ oc -n openshift-marketplace logs community-operators-fpzxq | grep -c "2025-07-25T12:43:36" 5996 $ oc -n openshift-marketplace logs community-operators-fpzxq | grep "2025-07-25T12:43:36" |grep -c "127.0.0.1" 4 $ oc -n openshift-marketplace logs community-operators-fpzxq | grep "2025-07-25T12:43:36" |grep -c "10.128.0.25" 2996 $ oc -n openshift-marketplace logs community-operators-fpzxq | grep "2025-07-25T12:43:36" |grep -c "10.129.0.19" 2996
[1]
$ oc -n openshift-marketplace logs community-operators-fpzxq | grep "2025-07-25T12:43:36"|egrep -c "GetPackage|GetBundleForChannel|ListPackages|ListBundles" 5992
[2]
$ oc -n openshift-operator-lifecycle-manager get pod packageserver-7bcccb4476-tr2k8 -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES packageserver-7bcccb4476-tr2k8 1/1 Running 0 37d 10.128.0.25 server.example.com <none> <none> $ oc -n openshift-operator-lifecycle-manager get pod packageserver-7bcccb4476-8j76m -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES packageserver-7bcccb4476-8j76m 1/1 Running 0 37d 10.129.0.19 server.example.com <none> <none>
[3]
$ oc -n openshift-operator-lifecycle-manager logs packageserver-7bcccb4476-8j76m |grep -c "2025-07-25T12:43:3" 2 $ oc -n openshift-operator-lifecycle-manager logs packageserver-7bcccb4476-tr2k8 |grep -c "2025-07-25T12:43:3" 2 $ oc -n openshift-operator-lifecycle-manager logs packageserver-7bcccb4476-tr2k8 |grep "2025-07-25T12:43:3" |head -1 2025-07-25T12:43:36.362730662Z time="2025-07-25T12:43:36Z" level=info msg="updating PackageManifest based on CatalogSource changes: {community-operators openshift-marketplace}" action="sync catalogsource" address="community-operators.openshift-marketplace.svc:50051" name=community-operators namespace=openshift-marketplace
