Dennis Periquet added a comment - 2024/03/27 10:39 AM - edited PR is merged and in 4.16.0-0.ci-2024-03-26-162223 (with passing payload). Looked at https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.16-upgrade-from-stable-4.15-e2e-azure-sdn-upgrade/1772660995244691456 which is one of the failing jobs (that ran for 3h30m) in https://prow.ci.openshift.org/view/gs/test-platform-results/logs/aggregated-azure-sdn-upgrade-4.16-minor-release-openshift-release-analysis-aggregator/1772660998604328960 and although the log file is still 31M, it does not contain the 3 logs removed by the PR. Looked at https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.16-upgrade-from-stable-4.15-e2e-azure-sdn-upgrade/1772660990232498176 (3h40m failure), 30M file but there are no event debug logs here either.

Dennis Periquet added a comment - 2024/03/15 1:15 PM - edited PR to remove event logs to reduce build-log.txt size: https://github.com/openshift/origin/pull/28654

Dennis Periquet added a comment - 2024/03/13 3:06 PM it's hard for me to believe that a 38M file can consume 4.6G. So I wonder if there's just some leaky process running on the build-log.txt file.

Dennis Periquet added a comment - 2024/03/12 6:57 PM This is the one that takes a long time to load and often results in chrome asking your to restart: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.15-e2e-metal-ipi-ovn-dualstack/1758641985364692992 When that job actually loads, you can see that the tab takes 4.6G: Upon loading that page, you will see the spinning Kubernetes logos on all the subframes.  If you compress all of them (except the one for build-log), you will see the one for the build-log.txt finishing last.  In this case, build-log.txt is 38M. As an aside, I mention in a slack thread that there are cases where a build-log.txt shows as 2.8M on the GCS bucket link (e.g., see  https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/28650/pull-ci-openshift-origin-master-e2e-metal-ipi-sdn/1767551429960011776 ).  But then, if you download that file, it's about 10x bigger. Anyway, I believe it's the big buid-log.txt files that slow down the prow job loads (much more so than the interval charts). If you run these lines on the build-log.txt file, you will see that certain log messages contribute about > 60% of the lines: $ cat build-log.txt |grep 'level=info msg= "processed event" ' |wc -l 21447 $ cat build-log.txt |grep 'level=info msg= "resulting interval locator" ' |wc -l 21447 $ cat build-log.txt |grep 'level=info msg= "resulting interval message" ' |wc -l 21447 $ cat build-log.txt |grep -v 'level=info msg= "processed event' |grep -v 'level=info msg=" resulting interval locator "' |grep -v 'level=info msg=" resulting interval message"' > small_build-log.txt $ ls -lh build-log.txt -rw-r--r-- 1 dperique wheel 32M Mar 12 14:42 build-log.txt $ ls -lh small_build-log.txt -rw-r--r-- 1 dperique wheel 3.3M Mar 12 14:54 small_build-log.txt (base) schubert:x dperique$ wc -l build-log.txt 93834 build-log.txt here is where the logs are generated.